diff options
Diffstat (limited to 'dev/multiarch.mdwn')
| -rw-r--r-- | dev/multiarch.mdwn | 180 |
1 files changed, 180 insertions, 0 deletions
diff --git a/dev/multiarch.mdwn b/dev/multiarch.mdwn new file mode 100644 index 0000000..d869a7d --- /dev/null +++ b/dev/multiarch.mdwn @@ -0,0 +1,180 @@ +[[!meta title="Multiarch Design"]] + +Earlier design proposals are documented in [[another_page|dev/multiarch/design]] +kept for historical purposes. + + +Background +========== + +ProteanOS is a self-hosting binary distribution that builds its own packages. +It is also an embedded distribution, intended to be suitable for small +resource-limited systems. It may not be practical to natively build all of +ProteanOS on real hardware for every architecture to be supported. Thus, it +will be preferable to cross-build all of ProteanOS's packages, making ProteanOS +a highly ambitious cross-built self-hosting binary distribution. + +Cross-compiling and cross-assembling self-contained code is a solved problem, +however programs typically use separately built libraries, which must be +installed at build time to be available for compiling and linking. Sometimes, +these dependency libraries will already be installed for use by the native +system. So it needs to be possible to "coinstall" different architecture builds +of each library. Package management and build tools also need to be able to +install dependency packages of the correct architecture; for example, libraries +must be of the *host* architecture (the architecture **for** which a package is +being built), but build utilities must be of the *build* architecture (the +architecture **on** which a package is being built). Further complicating +dependency resolution, such build utilities often themselves depend on +libraries, which must also be of the build architecture (not the host +architecture like libraries that are direct build-dependencies). These are the +two main problems in cross-building distribution packages: coinstallability and +dependencies. The solution is a design known as "multiarch". + +ProteanOS's multiarch design is inspired by, but different from, that +[specified][wuc-mas] and [documented][wdo-ma] in Debian and Ubuntu. + +ProteanOS's design solves the coinstallability and dependency problems without +*any* modifications to its package manager, opkg-lede. This is desirable +because opkg-lede is a tool used by many distributions separate from ProteanOS, +and its upstream maintainers (OpenWrt) would likely be unwilling to merge and +maintain invasive patches specific to a different distribution (making the +patches impossible to test upstream). + +[wdo-ma]: http://wiki.debian.org/Multiarch +[wuc-mas]: https://wiki.ubuntu.com/MultiarchSpec + + +Design +====== + +Coinstallability +---------------- + +opkg-lede doesn't allow multiple packages to provide the same file path, so, in +order to be coinstallable, library packages must install all of their files into +architecture-qualified locations. [Debian and Ubuntu][wuc-mas-arch-indep-files] +limit this requirement to only files whose contents differ between +architectures. This is accomplished by the package manager dpkg requiring all +packages of the same name to be the same version, verifying that shared files +have identical contents, and reference counting shared files. As explained +above, ProteanOS's multiarch design doesn't entail such modifications to the +package manager. + +opkg-lede also doesn't allow installation of multiple packages with the same +name and different architectures. The only way to solve this without modifying +the package manager is to qualify the names of coinstallable packages with +their host architectures. This will be done by appending a colon followed by +the host architecture to each coinstallable package name, e.g. `libc.6` becomes +`libc.6:amd64-linux-glibc`. Package names appear in the file system under +`/var/cache/opkg/archives/` and `/var/lib/opkg/info/`, so using a colon in +package names will prevent the use of restricted file systems such as FAT file +systems as the root file system. + +Dependencies +------------ + +opkg-lede allows packages from different architectures to be installed, but +there is no way to select the architecture to install other than blanket +priority values. Since libraries of different architectures will often need to +be installed during the same installation transaction, as described above, such +a blunt solution is insufficient. + +[Debian and Ubuntu][wuc-mas-control-fields] solve this by declaring the kinds of +dependencies each package is able to satisfy (based on the interfaces it +provides: executables and/or libraries). Again this requires modifications to +the package manager to affect dependency resolution. ProteanOS's design is +similar, but implemented in a way such that *dependent* binary packages are +responsible for declaring the architectures of *their dependencies* (which can +be done automatically at build time, so package maintainers can focus on +declaring how their packages satisfy dependencies, as in Debian and Ubuntu). + +One way to declare dependency semantics is by introducing a new single-purpose +control field like Debian's and Ubuntu's `Multi-Arch`. However, ProteanOS will +instead use [a control field that serves multiple purposes: +`Section`][spf-fields-bin]. `Section: lib` packages will be automatically made +coinstallable by opkbuild, by architecture-qualifying their names as described +above. Library dependencies generated by oh-shlibdeps (with [a +fix][oh-shlibdeps-host-libdirs] to be released) will automatically use the +architecture-qualified names. + +ProteanOS systems should have only native architecture feeds for sections +`boot`, `dev`, and `util` but may have multiple feeds for sections `dbg`, `lib`, +and `libdev`. (Sections `doc`, `locale`, and `share` are +architecture-independent.) + +opkg-lede's architecture priorities as described above can be useful in certain +specific ways, through a [wrapper script that adds an `-a`/`--host-architecture` +option][opkg-wrapper] (similar to that of apt-get). First, since multiple +`Section: dbg` feeds may be downloaded and requiring users to +architecture-qualify such packages would be undesirable, such packages could be +installed instead via `opkg -a amd64-linux-glibc foo-dbg` or even simply `opkg +install foo-dbg` for the native architecture. Similarly, `Section: libdev` +packages as build dependencies could be automatically installed for the host +architecture by prokit supplying an appropriate `-a` option, allowing +maintainers to continue listing unqualified package names in `Build-Depends` +instead of having to add `:${Host-Arch}` to every `Section: libdev` package in +`Build-Depends`. Therefore, only `Section: lib` packages need to have +architecture-qualified names (due to the coinstallability issue described +above), and neither users nor maintainers are expected to manually install or +list as dependencies such architecture-qualified packages. + +[wuc-mas-arch-indep-files]: https://wiki.ubuntu.com/MultiarchSpec#Architecture-independent_files_in_multiarch_packages +[wuc-mas-control-fields]: https://wiki.ubuntu.com/MultiarchSpec#Binary_package_control_fields +[spf-fields-bin]: http://specs.proteanos.com/spf-2.0/fields.html#fields-bin +[oh-shlibdeps-host-libdirs]: http://git.proteanos.com/opkhelper/opkhelper.git/commit/?id=c214c727f60b070aa502b20dcc1ad7e3ce8579f2 +[opkg-wrapper]: http://git.proteanos.com/pkg/opkg-lede.git/tree/opkg + + +Use Cases +========= + +**Run-time dependency on a utility:** Only the native architecture of the +utility dependency is available in the downloaded feed lists, and the dependency +is not architecture-qualified. The native architecture will be installed. + +**Run-time dependency on a library:** Multiple architectures of the library +dependency may be available, and the dependency must be qualified with the +native architecture. + +**Run-time dependency on a language extension:** Despite technically being a +shared object file, only the native architecture of the language extension +dependency is available in the downloaded feed lists, and the dependency is not +architecture-qualified. The language extension dependency in turn depends on a +language interpreter, which is also not architecture-qualified. The native +architecture of both will be installed. + +**Build-time dependency on a utility:** Only the build architecture of the +utility dependency is available in the downloaded feed lists, and the dependency +is not architecture-qualified. The build architecture will be installed. + +**Build-time dependency on a library:** Multiple architectures of the library +dependency are available, and the dependency must be qualified with the host +architecture. + +**Build-time dependency on a language extension:** Despite technically being a +shared object file, only the native architecture of the language extension +dependency is available in the downloaded feed lists, and the dependency is not +architecture-qualified. The language extension dependency in turn depends on a +language interpreter, which is also not architecture-qualified. The native +architecture of both will be installed. + + +Implementation +============== + +File System Hierarchy +--------------------- + +As explained above, to be coninstallable, library packages must install all of +their files in architecture-qualified locations. Debian and Ubuntu install +shared object files in `/usr/lib/<triplet>` or `/lib/<triplet>`, where +`<triplet>` is a [GNU system type][ac-systemtype]. Since multiple ProteanOS +architectures may share the same GNU system type, ProteanOS architecture names +are instead used in such library directories. + +ProteanOS's GNU Binutils and GCC packages have been patched and configured to +use multiarch library paths since their initial releases in 2014. opkhelper has +used multiarch library paths when configuring packages since version 3.0.0-beta1 +released in 2012. + +[ac-systemtype]: https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.69/html_node/System-Type.html |