diff options
Diffstat (limited to 'dev')
-rw-r--r-- | dev/archive/mirroring.mdwn | 3 | ||||
-rw-r--r-- | dev/multiarch.mdwn | 180 | ||||
-rw-r--r-- | dev/opkbuild.mdwn | 4 | ||||
-rw-r--r-- | dev/opkhelper.mdwn | 4 | ||||
-rw-r--r-- | dev/releases/1/packages.mdwn | 37 | ||||
-rw-r--r-- | dev/todo/install-bb-without-bb.mdwn | 86 | ||||
-rw-r--r-- | dev/wolfutil.mdwn | 48 |
7 files changed, 341 insertions, 21 deletions
diff --git a/dev/archive/mirroring.mdwn b/dev/archive/mirroring.mdwn index eb3f15c..deabb91 100644 --- a/dev/archive/mirroring.mdwn +++ b/dev/archive/mirroring.mdwn @@ -31,7 +31,8 @@ daily, so mirrors need not be updated more frequently than that. The [[archive_management_software|dev/pro-archman]] is [configured][arch-conf] to keep unreferenced files in the pool for one day, so mirrors must be updated at least once a day. Otherwise, during the update, some feed index files may -reference deleted files. +reference deleted files. Updating at least twice daily is recommended, in case +the synchronization source is ever temporarily unreachable. Serving the archive mirror at `/pub/proteanos` over HTTP and FTP is recommended, but any path and either protocol may be used. HTTP Strict Transport Security diff --git a/dev/multiarch.mdwn b/dev/multiarch.mdwn new file mode 100644 index 0000000..d869a7d --- /dev/null +++ b/dev/multiarch.mdwn @@ -0,0 +1,180 @@ +[[!meta title="Multiarch Design"]] + +Earlier design proposals are documented in [[another_page|dev/multiarch/design]] +kept for historical purposes. + + +Background +========== + +ProteanOS is a self-hosting binary distribution that builds its own packages. +It is also an embedded distribution, intended to be suitable for small +resource-limited systems. It may not be practical to natively build all of +ProteanOS on real hardware for every architecture to be supported. Thus, it +will be preferable to cross-build all of ProteanOS's packages, making ProteanOS +a highly ambitious cross-built self-hosting binary distribution. + +Cross-compiling and cross-assembling self-contained code is a solved problem, +however programs typically use separately built libraries, which must be +installed at build time to be available for compiling and linking. Sometimes, +these dependency libraries will already be installed for use by the native +system. So it needs to be possible to "coinstall" different architecture builds +of each library. Package management and build tools also need to be able to +install dependency packages of the correct architecture; for example, libraries +must be of the *host* architecture (the architecture **for** which a package is +being built), but build utilities must be of the *build* architecture (the +architecture **on** which a package is being built). Further complicating +dependency resolution, such build utilities often themselves depend on +libraries, which must also be of the build architecture (not the host +architecture like libraries that are direct build-dependencies). These are the +two main problems in cross-building distribution packages: coinstallability and +dependencies. The solution is a design known as "multiarch". + +ProteanOS's multiarch design is inspired by, but different from, that +[specified][wuc-mas] and [documented][wdo-ma] in Debian and Ubuntu. + +ProteanOS's design solves the coinstallability and dependency problems without +*any* modifications to its package manager, opkg-lede. This is desirable +because opkg-lede is a tool used by many distributions separate from ProteanOS, +and its upstream maintainers (OpenWrt) would likely be unwilling to merge and +maintain invasive patches specific to a different distribution (making the +patches impossible to test upstream). + +[wdo-ma]: http://wiki.debian.org/Multiarch +[wuc-mas]: https://wiki.ubuntu.com/MultiarchSpec + + +Design +====== + +Coinstallability +---------------- + +opkg-lede doesn't allow multiple packages to provide the same file path, so, in +order to be coinstallable, library packages must install all of their files into +architecture-qualified locations. [Debian and Ubuntu][wuc-mas-arch-indep-files] +limit this requirement to only files whose contents differ between +architectures. This is accomplished by the package manager dpkg requiring all +packages of the same name to be the same version, verifying that shared files +have identical contents, and reference counting shared files. As explained +above, ProteanOS's multiarch design doesn't entail such modifications to the +package manager. + +opkg-lede also doesn't allow installation of multiple packages with the same +name and different architectures. The only way to solve this without modifying +the package manager is to qualify the names of coinstallable packages with +their host architectures. This will be done by appending a colon followed by +the host architecture to each coinstallable package name, e.g. `libc.6` becomes +`libc.6:amd64-linux-glibc`. Package names appear in the file system under +`/var/cache/opkg/archives/` and `/var/lib/opkg/info/`, so using a colon in +package names will prevent the use of restricted file systems such as FAT file +systems as the root file system. + +Dependencies +------------ + +opkg-lede allows packages from different architectures to be installed, but +there is no way to select the architecture to install other than blanket +priority values. Since libraries of different architectures will often need to +be installed during the same installation transaction, as described above, such +a blunt solution is insufficient. + +[Debian and Ubuntu][wuc-mas-control-fields] solve this by declaring the kinds of +dependencies each package is able to satisfy (based on the interfaces it +provides: executables and/or libraries). Again this requires modifications to +the package manager to affect dependency resolution. ProteanOS's design is +similar, but implemented in a way such that *dependent* binary packages are +responsible for declaring the architectures of *their dependencies* (which can +be done automatically at build time, so package maintainers can focus on +declaring how their packages satisfy dependencies, as in Debian and Ubuntu). + +One way to declare dependency semantics is by introducing a new single-purpose +control field like Debian's and Ubuntu's `Multi-Arch`. However, ProteanOS will +instead use [a control field that serves multiple purposes: +`Section`][spf-fields-bin]. `Section: lib` packages will be automatically made +coinstallable by opkbuild, by architecture-qualifying their names as described +above. Library dependencies generated by oh-shlibdeps (with [a +fix][oh-shlibdeps-host-libdirs] to be released) will automatically use the +architecture-qualified names. + +ProteanOS systems should have only native architecture feeds for sections +`boot`, `dev`, and `util` but may have multiple feeds for sections `dbg`, `lib`, +and `libdev`. (Sections `doc`, `locale`, and `share` are +architecture-independent.) + +opkg-lede's architecture priorities as described above can be useful in certain +specific ways, through a [wrapper script that adds an `-a`/`--host-architecture` +option][opkg-wrapper] (similar to that of apt-get). First, since multiple +`Section: dbg` feeds may be downloaded and requiring users to +architecture-qualify such packages would be undesirable, such packages could be +installed instead via `opkg -a amd64-linux-glibc foo-dbg` or even simply `opkg +install foo-dbg` for the native architecture. Similarly, `Section: libdev` +packages as build dependencies could be automatically installed for the host +architecture by prokit supplying an appropriate `-a` option, allowing +maintainers to continue listing unqualified package names in `Build-Depends` +instead of having to add `:${Host-Arch}` to every `Section: libdev` package in +`Build-Depends`. Therefore, only `Section: lib` packages need to have +architecture-qualified names (due to the coinstallability issue described +above), and neither users nor maintainers are expected to manually install or +list as dependencies such architecture-qualified packages. + +[wuc-mas-arch-indep-files]: https://wiki.ubuntu.com/MultiarchSpec#Architecture-independent_files_in_multiarch_packages +[wuc-mas-control-fields]: https://wiki.ubuntu.com/MultiarchSpec#Binary_package_control_fields +[spf-fields-bin]: http://specs.proteanos.com/spf-2.0/fields.html#fields-bin +[oh-shlibdeps-host-libdirs]: http://git.proteanos.com/opkhelper/opkhelper.git/commit/?id=c214c727f60b070aa502b20dcc1ad7e3ce8579f2 +[opkg-wrapper]: http://git.proteanos.com/pkg/opkg-lede.git/tree/opkg + + +Use Cases +========= + +**Run-time dependency on a utility:** Only the native architecture of the +utility dependency is available in the downloaded feed lists, and the dependency +is not architecture-qualified. The native architecture will be installed. + +**Run-time dependency on a library:** Multiple architectures of the library +dependency may be available, and the dependency must be qualified with the +native architecture. + +**Run-time dependency on a language extension:** Despite technically being a +shared object file, only the native architecture of the language extension +dependency is available in the downloaded feed lists, and the dependency is not +architecture-qualified. The language extension dependency in turn depends on a +language interpreter, which is also not architecture-qualified. The native +architecture of both will be installed. + +**Build-time dependency on a utility:** Only the build architecture of the +utility dependency is available in the downloaded feed lists, and the dependency +is not architecture-qualified. The build architecture will be installed. + +**Build-time dependency on a library:** Multiple architectures of the library +dependency are available, and the dependency must be qualified with the host +architecture. + +**Build-time dependency on a language extension:** Despite technically being a +shared object file, only the native architecture of the language extension +dependency is available in the downloaded feed lists, and the dependency is not +architecture-qualified. The language extension dependency in turn depends on a +language interpreter, which is also not architecture-qualified. The native +architecture of both will be installed. + + +Implementation +============== + +File System Hierarchy +--------------------- + +As explained above, to be coninstallable, library packages must install all of +their files in architecture-qualified locations. Debian and Ubuntu install +shared object files in `/usr/lib/<triplet>` or `/lib/<triplet>`, where +`<triplet>` is a [GNU system type][ac-systemtype]. Since multiple ProteanOS +architectures may share the same GNU system type, ProteanOS architecture names +are instead used in such library directories. + +ProteanOS's GNU Binutils and GCC packages have been patched and configured to +use multiarch library paths since their initial releases in 2014. opkhelper has +used multiarch library paths when configuring packages since version 3.0.0-beta1 +released in 2012. + +[ac-systemtype]: https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.69/html_node/System-Type.html diff --git a/dev/opkbuild.mdwn b/dev/opkbuild.mdwn index ef489c4..9e45e00 100644 --- a/dev/opkbuild.mdwn +++ b/dev/opkbuild.mdwn @@ -19,7 +19,7 @@ opkbuild is similar in function to the [development library][libdpkg-perl] and Downloading =========== -The current version of opkbuild is [4.1.4][rel-ann]. +The current version of opkbuild is [4.2.0][rel-ann]. opkbuild can be found on the ProteanOS files site by [HTTP][dl-http] or [FTP][dl-ftp]. @@ -27,7 +27,7 @@ opkbuild can be found on the ProteanOS files site by [HTTP][dl-http] or opkbuild is maintained in a [Git repository][repo-http], which can be cloned from `git://git.proteanos.com/opkbuild/opkbuild.git`. -[rel-ann]: http://lists.proteanos.com/proteanos-dev/2019/08/msg00001.html +[rel-ann]: http://lists.proteanos.com/proteanos-dev/2020/07/msg00000.html [dl-http]: http://files.proteanos.com/pub/opkbuild/ [dl-ftp]: ftp://files.proteanos.com/pub/opkbuild/ [repo-http]: http://git.proteanos.com/opkbuild/opkbuild.git/ diff --git a/dev/opkhelper.mdwn b/dev/opkhelper.mdwn index 9d652c5..47bcd49 100644 --- a/dev/opkhelper.mdwn +++ b/dev/opkhelper.mdwn @@ -14,7 +14,7 @@ Debian. Downloading =========== -The current version of opkhelper is [3.1.2][rel-ann]. +The current version of opkhelper is [3.1.3][rel-ann]. opkhelper can be found on the ProteanOS files site by [HTTP][dl-http] or [FTP][dl-ftp]. @@ -22,7 +22,7 @@ opkhelper can be found on the ProteanOS files site by [HTTP][dl-http] or opkhelper is maintained in a [Git repository][repo-http], which can be cloned from `git://git.proteanos.com/opkhelper/opkhelper.git`. -[rel-ann]: http://lists.proteanos.com/proteanos-dev/2019/03/msg00003.html +[rel-ann]: http://lists.proteanos.com/proteanos-dev/2020/07/msg00001.html [dl-http]: http://files.proteanos.com/pub/opkhelper/ [dl-ftp]: ftp://files.proteanos.com/pub/opkhelper/ [repo-http]: http://git.proteanos.com/opkhelper/opkhelper.git/ diff --git a/dev/releases/1/packages.mdwn b/dev/releases/1/packages.mdwn index 13b03d7..02f6810 100644 --- a/dev/releases/1/packages.mdwn +++ b/dev/releases/1/packages.mdwn @@ -11,6 +11,7 @@ packages: base-files 1.0.3 bc 1.07.1-1 binutils 2.32+sip1-1 + bison 3.4.1-1 build-essential 1.2 busybox 1.21.1-13 bzip2 1.0.6-3 @@ -32,12 +33,14 @@ packages: fbida 2.09-2 file 5.37-3 flashrom 1.1~rc1-1 + flex 2.6.4-1 fontconfig 2.11.1-1 freetype 2.5.3-1 fss 1.0 gawk 5.0.0-1 gcc-8 8.3.0+sip1-1 gcc-defaults 1.1 + gdb 8.3+sip1-1 gettext 0.18.3.2-1 gettext-tiny 0.3.1 gmake 4.2.1-1 @@ -58,47 +61,51 @@ packages: libpng12 1.2.51-2 libsigsegv 2.12-1 libtomcrypt 1.18.2-1 - libtommath.1 1.1.0-1 libtommath 0.42.0-1 + libtommath.1 1.1.0-1 libtool 2.4.2-2 libubox 0+git20190227.eeef7b5-1 libunistring 0.9.10-1 libvorbis 1.3.4-1 lilo 24.2-1 linux-libre 3.10.60~gnu-1 - m4 1.4.17-2 + m4 1.4.18-1 mpc 1.1.0-1 mpfr 4.0.2-1 mplus-fonts 058-2 - ncurses 6.1+20190623-1 + ncurses 6.1+20190706-1 npth 1.6-1 + nss-certdata 3.54-2 open-ath9k-htc-firmware 1.4~git20141115.146bff1-1 - opkbuild 4.1.2-1 + opkbuild 4.2.0-1 opkg 0.2.4-3 opkg-lede 0+git20190131.d4ba162-6 - opkhelper-3.0 3.1.2-2 + opkhelper-3.0 3.1.3-1 opkhelper-default 3.0.0 patch 2.7.6-1 pciids 2019.06.17-1 pciutils 3.6.2-1 pcre 8.35-1 pkgconf 0.9.6-1 - platconf-pkg-tools 2.1.1 + platconf-pkg-tools 3.0.0 popt 1.16+sip1-1 + proquivs 1.0 proteanos-branding 0.3 pth 2.0.7-1 - readline.8 8.0-1 readline 6.3-1 + readline.8 8.0-1 strace 4.8-1 tcl8.6 8.6.9+sip1-2 tzdb 2019a-1 usign 0+git20161022.3e6648b-1 + wolfssl.24 4.4.0-3 + wolfutil 1.0.0-2 wpa-supplicant 2.2-2 xz 5.2.4-1 zlib 1.2.11+sip1-1 ------------------------------------------------------------ - Source packages: 91 - Binary packages: 369 + Source packages: 98 + Binary packages: 395 The above list was generated by running the following shell script: @@ -108,9 +115,6 @@ The above list was generated by running the following shell script: verw=33 base_url='http://files.proteanos.com/pub/proteanos/feeds/dev/trunk' - LF=' - ' - # Print header i=0 l=$(($pkgw + $verw + 2)) @@ -135,17 +139,18 @@ The above list was generated by running the following shell script: ;; esac done <<EOF - $(wget -q -O - "${base_url}/src/all/base/Packages") + $(wget -q -O - "${base_url}/src/all/base/Packages" \ + "${base_url}/src/all/src/Packages") EOF # Collect binary package names bins='' - for aps in $(wget -q -O - "${base_url}/Manifest" | grep -Fv 'src/all/base'); do - bins="${bins}${LF}$(wget -q -O - "${base_url}/${aps}/Packages" | \ + for aps in $(wget -q -O - "${base_url}/Manifest" | grep -Fv 'src/all/'); do + bins="${bins} $(wget -q -O - "${base_url}/${aps}/Packages" | \ sed -n 's/^Package: //p;')" done # Print totals printf '%s\n' "${tblline}" printf 'Source packages: %d\n' "${srcs}" - printf 'Binary packages: %d\n' "$(printf '%s\n' "${bins}" | sort -u | wc -l)" + printf 'Binary packages: %d\n' "$(printf '%s\n' ${bins} | sort -u | wc -l)" diff --git a/dev/todo/install-bb-without-bb.mdwn b/dev/todo/install-bb-without-bb.mdwn new file mode 100644 index 0000000..4bb90ff --- /dev/null +++ b/dev/todo/install-bb-without-bb.mdwn @@ -0,0 +1,86 @@ +[[!meta title="Installing BusyBox Without BusyBox"]] + +**or, Adventures in Regressions and Crazy Ideas** + +--- + +**This work is now done.** + +**The rest of this page is kept only for reference and historical purposes.** + +--- + +Two Problems +============ + +`opkg-lede` and the Outsourced Inflation +---------------------------------------- + +`opkg-lede` [now relies][opkg-lede-dd4c78a] on an external `gzip` executable. +When upgrading a package, it tries to execute `gzip` to unpack maintainer +scripts and data files [after removing][opkg_install_pkg] the old version of the +package. When the package being upgraded is `busybox`, which provides `gzip`, +`opkg` crashes and leaves behind a completely broken system. + +To reiterate, `opkg-lede` is (and has been for as long as it has been in +ProteanOS) **currently unable to upgrade `busybox`**, due to its own design. + +[opkg-lede-dd4c78a]: https://git.openwrt.org/?p=project/opkg-lede.git;a=commitdiff;h=dd4c78aa88efd3b9cf516030937c684814df7962 +[opkg_install_pkg]: https://git.openwrt.org/?p=project/opkg-lede.git;a=blob;f=libopkg/opkg_install.c;hb=d4ba162#l1520 + +The Amazing Alternatives of `busybox` +------------------------------------- + +Instead of [adding alternatives][alts-log] one at a time, [it would be +nice][bb-ua] to manage all of `busybox`'s utility links using +`update-alternatives`. Unfortunately, the [maintainer script][bb.pi] that runs +`update-alternatives` (as well as the [`update-alternatives` script][u-a] +itself) needs a few `busybox` utilities (`/bin/sh` for a start). Thus, after an +older version of `busybox` is deconfigured, a newer version can't be configured. +`busybox` also can't be configured when first unpacked by `prokit`. + +[alts-log]: http://git.proteanos.com/pkg/busybox.git/log/alternatives +[bb-ua]: http://git.proteanos.com/pkg/busybox.git/commit/?id=1558d05c62ed90f7d93a14533541c33896b7b733 +[bb.pi]: http://git.proteanos.com/pkg/busybox.git/tree/busybox.pkg/postinst +[u-a]: https://git.openwrt.org/?p=project/opkg-lede.git;a=blob;f=utils/update-alternatives.in;hb=d4ba162 + +One Solution +============ + +The next version of `src:opkg-lede` (likely 0+git20190131.d4ba162-7) already +will have a [wrapper script called `opkg`][opkg] that executes `opkg-cl` (the +actual binary executable). The script can be extended to first save a copy of +`/bin/busybox` to an empty temporary directory, generate utility links to it in +the directory, and add the directory to `PATH`. Until all of this setup is +complete, the wrapper script must be careful to execute utilities as `busybox +$util` (a full path to `busybox` won't be necessary, as the usual `/bin/busybox` +will be available at that point). This includes the interpreter line, which +will need to be changed to `#!/bin/busybox sh`. `ash` built-in utilities can be +used as usual. + +The `busybox` package can't use `update-alternatives` to manage all of its +utility links until an `opkg-lede` with this wrapper script functionality is +uploaded. This is because, even if `busybox`'s `postinst` script is modified to +use the `busybox` executable directly, other packages' maintainer scripts may be +executed first, while `busybox` is unpacked but not yet configured (and +therefore utility links are not yet created). + +Once this new `opkg-lede` package is uploaded and `busybox` uses +`update-alternatives` for everything, `busybox` would need to declare that it +`Breaks` older versions of `opkg-lede`, to ensure that `opkg-lede` will preserve +a copy of the `busybox` executable and its utility links. However, this is +still insufficient, as the version of `opkg` that is *running* must have already +set up this environment before `busybox` is upgraded. `busybox`'s `prerm` +script will need to check for this somehow and abort the upgrade if `opkg` is +too old to have prepared for a safe upgrade. One solution is for the `opkg` +script to set an `OPKG_BUSYBOX_SAVED` environment variable. + +The `opkg` script of the `opkg-lede` package will need to read `busybox`'s list +of alternatives links. This is currently located at +`/usr/share/busybox/alternatives`. [The file's format][alts] will change when +`busybox` uses `update-alternatives` for everything: the last two columns (name +and priority) will be dropped. `opkg` however can simply read the first column, +so it will maintain compatibility. + +[opkg]: http://git.proteanos.com/pkg/opkg-lede.git/tree/opkg +[alts]: http://git.proteanos.com/pkg/busybox.git/tree/alternatives?id=e37f03311d934a8f33a9ff31db47b18bbf0be555 diff --git a/dev/wolfutil.mdwn b/dev/wolfutil.mdwn new file mode 100644 index 0000000..1fba2fe --- /dev/null +++ b/dev/wolfutil.mdwn @@ -0,0 +1,48 @@ +[[!meta title="Unofficial wolfSSL Utility"]] + +wolfutil is a command-line tool for the wolfSSL embedded (SSL/)TLS +library. It is similar to the OpenSSL command-line tool, providing only +a limited `s_client` TLS client command. The primary goal is to provide +enough functionality for use by BusyBox's wget applet. + +wolfutil's `s_client` command is designed to be stronger by default than +OpenSSL's, as the latter is only meant to be a "diagnostic tool". That +is, the following command: + + $ wolfutil s_client -quiet -connect ${host}:${port} \ + > -servername ${servername} + +is roughly equivalent to the more complicated: + + $ openssl s_client -quiet -connect ${host}:${port} \ + > -servername ${servername} -verify 9 -verify_return_error \ + > -no_ssl3 -no_tls1 -no_tls1_1 + +This utility is in no way authored by or affiliated with wolfSSL Inc. or +its contributors. + + +Downloading +=========== + +The current version of wolfutil is [1.0.0][rel-ann]. + +wolfutil can be found on the ProteanOS files site by [HTTP][dl-http] or +[FTP][dl-ftp]. + +wolfutil is maintained in a [Git repository][repo-http], which can be cloned +from `git://git.proteanos.com/wolfutil/wolfutil.git`. + +[rel-ann]: http://lists.proteanos.com/proteanos-dev/2020/08/msg00000.html +[dl-http]: http://files.proteanos.com/pub/wolfutil/ +[dl-ftp]: ftp://files.proteanos.com/pub/wolfutil/ +[repo-http]: http://git.proteanos.com/wolfutil/wolfutil.git/ + + +Mailing List +============ + +Bug reports, feature requests, and patches for wolfutil are welcome on the +[ProteanOS development mailing list][proteanos-dev]. + +[proteanos-dev]: http://lists.proteanos.com/proteanos-dev/ |