path: root/dev/todo.html

<!--#set var="title" value="Development Projects" -->
<!--#set var="pagetitle" value="Development Projects" -->
<!--#include virtual="../includes/header.html" -->
<p>The following is an overview of development projects that need to be done before the operating system can be considered ready for production use.</p>
<h2 id="boot-sequencing">Boot Sequencing</h2>
<p>Currently, service init scripts are provided by the monolithic <code>initscripts</code> binary package. These init scripts are executed in the lexicographic order of the symbolic links matching <code>/etc/rc.d/S*</code> that target them.  The names of these symbolic links are currently hardcoded in <a href="http://git.os.pehjota.net/pkg/basefiles.git/tree/build?id=0ae24516#n43">the <code>build</code> makefile</a> of the <code>basefiles</code> source package.  This monolithic packaging and hardcoding of link names is a temporary and poor technical solution that doesn't scale with the number and selection of services that can be installed on a system.  Init scripts should instead be provided by the packages that provide the relevant system services.  The order in which init scripts are executed should be determined by dynamic boot sequencing based on inter-service dependency metadata.</p>
<p>This boot sequencing can be done when the system boots or after a new system service is installed.  For reference, NetBSD <a href="http://www.netbsd.org/docs/guide/en/chap-rc.html#chap-rc-rcorder">uses a program called "rcorder"</a> to determine a boot sequence at boot time.  Many GNU/Linux distributions follow (to some degree) the Linux Standard Base (LSB) specification, which defines <a href="http://refspecs.linuxbase.org/LSB_3.1.0/LSB-Core-generic/LSB-Core-generic/initscrcomconv.html">"Comment Conventions" for dependency metadata</a> and <a href="http://refspecs.linuxbase.org/LSB_3.1.0/LSB-Core-generic/LSB-Core-generic/initsrcinstrm.html">a method for installing sequentially-named symbolic links</a>.  Conforming implementations perform boot sequencing at the time of service installation.  Because boot sequencing at boot time can slow down system booting, it is better to perform boot sequencing at install time.</p>
<p>Thus, generally speaking, the solution to be adopted in this system is to make packages that provide system services also include the necessary init scripts (installed in <code>/etc/init.d</code>), to include inter-service dependency metadata in init scripts, and to use a tool at the time of service package installation to generate sequentially-named symoblic links in <code>/etc/rc.d</code>.</p>
<p>An obvious boot sequencing tool is <a href="http://svn.savannah.nongnu.org/viewvc/insserv/trunk/?root=sysvinit">"insserv"</a> maintained by Werner Fink and used by Debian and openSUSE.  However, this C program (in compliance with the LSB) assumes the use of runlevels.  This operating system uses the init daemon of BusyBox, which <a href="http://git.busybox.net/busybox/tree/init/init.c?id=3da46c81#n1221">doesn't support runlevels</a>.  Therefore, we'll need to either modify insserv to work without runlevels or write our own tool for installing symbolic links to init scripts.</p>
<p>Additionally, we need to decide how completely we'll conform, if at all, with the LSB in this area.</p>
<p>Hopefully, this can get done by September 2012.</p>
<h2 id="ma">Multiarch</h2>
<p>Multiarch refers to the ability to install and use packages built for non-native architectures.  It is currently being <a href="http://wiki.debian.org/Multiarch">documented and implemented</a> in Debian and Ubuntu.  Multiarch is useful for this distribution because it makes cross compiling easy (see "Package Cross Building Tool" and "Multiarch Cross Toolchain Packages" below).</p>
<p>Simply speaking, there are six <a href="https://wiki.ubuntu.com/MultiarchSpec#Design">aspects of a multiarch implementation</a>:</p>
<ul>
	<li>
		<h4>Filesystem Hierarchy</h4>
		<p>To accomodate simultaneous installation of shared libraries built for different architectures, library paths are suffixed with a directory name that identifies a particular architecture.  For this distribution, that directory name will be the name of the binary architecture; for example, library paths for the <code>cortexa8-linux-eglibc</code> architecture will be <code>/lib/cortexa8-linux-eglibc</code> and <code>/usr/lib/cortexa8-linux-eglibc</code>.</p>
		<p>The toolchain (especially the dynamic linker) needs to be configured to use these library paths.</p>
	</li>
	<li>
		<h4>A <code>Multi-Arch</code> Control Field</h4>
		<p>A <code>Multi-Arch</code> control field specifies how a package can satisfy dependencies of other packages.  A value of <code>same</code> means that a package can only satisfy dependencies of packages built for the architecture for which it was built.  This is useful for shared libraries, which can only be dynamically linked against binary objects built for the same architecture.  A value of <code>foreign</code> means that a package can satisfy dependencies of packages built for any architecture.  This is useful for utility programs, which can be used by software built for any architecture.</p>
		<p>The package manager (opkg in our case) and package building tools (specifically oh-checkbuilddeps of opkhelper) must be able to understand and use this field in resolving package dependencies.</p>
	</li>
	<li>
		<h4>Architecture Specification in Package Relationships</h4>
		<p>Control fields like <code>Depends</code> are extended to support an architecture specification of either <code>any</code> or <code>same</code>, allowing a package to specify whether or not it needs a package of the same architecture.  The dependent package has a <code>Multi-Arch</code> field with value <code>allowed</code>.</p>
		<p>The package manager and package building tools must be able to understand and use this architecture syntax in relationship fields.</p>
	</li>
	<li>
		<h4>Handling of Packages with <code>Architecture: all</code></h4>
		<p>Dependence on packages installable and usable on any architecture (especially considering opkg's ability to install packages of multiple architectures) must be researched.</p>
	</li>
	<li>
		<h4>Non-Library Files in Library Packages</h4>
		<p>Library packages provide files outside of system library paths, such as configuration and package documentation files.  A solution must be designed to allow a package built for one architecture to be co-installable with the same package built for a different architecture (or to determine if they are co-installable at all).</p>
	</li>
	<li>
		<h4>Ability to Install Packages Built for Non-Native Architectures</h4>
		<p>The package manager must be able to install packages built for multiple architectures (the whole point of multiarch).  Fortunately for us, opkg already handles this.  The <code>arch</code> option in <code>opkg.conf</code> <a href="http://wiki.openwrt.org/doc/techref/opkg#adjust.architectures">allows the user to specify architectures</a> for which packages can be installed.</p>
	</li>
</ul>
<p>In summary, there is much design work to be done, opkg and opkhelper must be modified to support multiarch, and certain packages will need to be built to handle multiarch library paths.  Of course Debian is a great reference implementation, but there still remains much original work to be done.</p>
<h2 id="installation-bootstrap-tool">Installation Bootstrap Tool</h2>
<p>A tool similar to <a href="http://anonscm.debian.org/gitweb/?p=d-i/debootstrap.git;a=tree">debootstrap</a> of Debian needs to be written to bootstrap the installation of a basic system.  It can be used for building packages (see "Package Cross Building Tool" below) or installing the operating system on hardware targets.</p>
<p>Basically, the tool would fetch from the package archive the index of packages, determine which packages need to be installed, download each package, and unpack each package.  Since the package manager may not be available, the tool must handle dependency resolution and package unpacking on its own.</p>
<p>To be determined is how the "second stage" of the installation &ndash; the execution of package maintainer scripts (<code>preinst</code> and <code>postinst</code>) to complete the configuration of each package &ndash; will be done.  At least most of the time, this tool will be used to install packages built for an architecture that differs from the architecture on which the tool is run; therefore utilities used by maintainer scripts may not be executable.  In this situation, debootstrap leaves behind a copy of itself in the installed system to be executed on the target architecture.  Such a solution might not work for this tool, because nothing can be executed on the target architecture until the installed system is booted, and the installed system shouldn't be booted until after the packages are configured.</p>
<p>We can't use debootstrap, since the formats of our binary packages and package archives differ slightly from those of Debian.  But we can model our tool after debootstrap or even just fork debootstrap.</p>
<p>If written portably (i.e. in conformance with POSIX.1), this tool could be used to make base system images on any UNIX-like operating system with an implementation of tar.  On any operating system that also has a chroot program, this tool can be used with the package cross building tool described below to build packages for this distribution.  Therefore, these tools can be thought of as a "Software Development Kit" ("SDK") for the distribution, usable on any capable development system.</p>
<p>Hopefully, this tool can be done by October 2012.</p>
<h2 id="pkg-x-build-tool">Package Cross Building Tool</h2>
<p>A tool similar to pbuilder and sbuild of Debian needs to be written to build packages within a chroot environment containing a base system installed by the installation bootstrap tool.  It needs to support cross building of packages using multiarch cross toolchains.</p>
<h2 id="ma-x-toolchains">Multiarch Cross Toolchain Packages</h2>
<p>Needed are packages of toolchain components (e.g. GCC and EGLIBC) that use multiarch library paths.</p>
<p>There is currently a <a href="http://wiki.debian.org/SummerOfCode2012/Projects#Multiarch_Cross-Toolchains">Google Summer of Code 2012 project</a> to <a href="http://wiki.debian.org/SummerOfCode2012/StudentApplications/ThibautGirka">develop such packages for Debian</a>.</p>
<h2 id="packaging">Packaging</h2>
<p>There are always more source packages to be made.  Software that should be packaged soon includes:</p>
<ul>
	<li>
		<h4><a href="http://matt.ucc.asn.au/dropbear/dropbear.html">Dropbear</a></h4>
		<p>Dropbear is a small SSH server and client, in many ways compatible with OpenSSH.</p>
		<p>In addition to the basic packaging work, there is work to be done on a service script (just a simple shell script in <code>/etc/init.d</code>) and <code>postinst</code> and <code>postrm</code> maintainer scripts to generate and delete the SSH host key pair.</p>
	</li>
	<li>
		<h4><a href="https://www.gnu.org/software/autoconf/">GNU Autoconf</a></h4>
		<p>GNU Autoconf generates <code>configure</code> scripts that are used to configure software packages for building.</p>
	</li>
	<li>
		<h4><a href="https://www.gnu.org/software/automake/">GNU Automake</a></h4>
		<p>GNU Automake generates <code>Makefile.in</code> files that are used to build software packages.</p>
	</li>
	<li>
		<h4><a href="https://www.gnu.org/software/m4/">GNU M4</a></h4>
		<p>GNU M4 is a macro processor, notably used by GNU Autoconf.</p>
	</li>
	<li>
		<h4><a href="http://www.perl.org/">Perl 5</a></h4>
		<p>Perl 5 is a language interpreter, especially popular in systems administration and software build and installation systems.</p>
		<p>Unmodified Perl 5 source is impossible to cross build without executing software on the host system (in GNU Autoconf terms, the system for which the package is built).  See <a href="https://lists.debian.org/debian-embedded/2012/06/msg00011.html">this mailing list thread</a> for more information.  We will need to modify Perl's build system a bit before we can build a package.</p>
	</li>
	<li>
		<h4><a href="http://gmplib.org/">GNU Multiple Precision Arithmetic Library (GMP)</a></h4>
		<p>GMP is a free library for arbitrary precision arithmetic, operating on signed integers, rational numbers, and floating point numbers.  It is used by GCC.</p>
	</li>
	<li>
		<h4><a href="http://www.mpfr.org/">GNU MPFR</a></h4>
		<p>MPFR is a C library for multiple-precision floating-point computations with correct rounding.</p>
	</li>
	<li>
		<h4><a href="http://www.multiprecision.org/index.php?prog=mpc">GNU MPC</a></h4>
		<p>MPC is a C library for the arithmetic of complex numbers with arbitrarily high precision and correct rounding of the result.</p>
	</li>
	<li>
		<h4><a href="https://www.gnu.org/software/binutils/">GNU Binutils</a></h4>
		<p>GNU Binutils is a collection of binary utilities, including a linker and assembler.</p>
		<p>Binutils and GCC are part of the <a href="#ma-x-toolchains">multiarch cross toolchain</a> project.</p>
	</li>
	<li>
		<h4><a href="http://gcc.gnu.org/">GNU Compiler Collection (GCC)</a></h4>
		<p>GCC is an optimizing compiler with frontends and libraries for a wide range of languages.</p>
		<p>Binutils and GCC are part of the <a href="#ma-x-toolchains">multiarch cross toolchain</a> project.</p>
	</li>
	<li>
		<h4><a href="https://www.gnu.org/software/make/">GNU Make</a></h4>
		<p>GNU Make is a tool that automatically builds software packages.</p>
	</li>
	<li>
		<h4><a href="http://www.denx.de/wiki/U-Boot">DAS U-Boot</a></h4>
		<p>U-Boot is a bootloader used on many embedded computers, including the BeagleBoard-xM.</p>
	</li>
</ul>
<h2 id="opkg-libbb">Removing libbb from opkg</h2>
<p>opkg includes and is statically linked with <a href="https://code.google.com/p/opkg/source/browse/trunk/libbb">a copy of libbb</a>, BusyBox's helper library.  Of course, BusyBox is also linked against its own copy of libbb.  This is a bad idea in general, since duplicated code makes maintenance more difficult and is often subject to bit rot (resulting in long-standing bugs and security problems).  It's an even larger problem (no pun intended) in embedded systems, where saving space is important.</p>
<h4>The Good News</h4>
<p>libbb can be built as a shared library by setting the <code>CONFIG_BUILD_LIBBUSYBOX</code> option to <code>y</code> in BusyBox's configuration.  So, we can make <code>libbusybox.N</code> and <code>libbusybox.N-dev</code> binary packages that provide this shared library and its header files, modify opkg to dynamically link against this library (instead of statically linking against its own copy), and add dependency information to the <code>opkg</code> source package and its binary packages.</p>
<h4>The Bad News</h4>
<p>The copy of libbb in opkg hasn't been updated from its upstream source since at least 2008, when opkg development began from the old ipkg sources.  Three months later, <a href="https://groups.google.com/group/opkg-devel/browse_thread/thread/fe5c620bc339ba63">an attempt</a> was made by Koen Kooi to update the copy of libbb, but it was determined that BusyBox had changed significantly since the code was added to ipkg.  Things have most likely gotten much worse in the more than three years since then.</p>
<p>To make matters slightly worse, many of the libbb files have been patched in opkg.  Any files in <a href="https://code.google.com/p/opkg/source/browse/trunk/libbb">/trunk/libbb</a> newer than revision 3 have opkg modifications.</p>
<h4>The Plan</h4>
<ul>
	<li>Build a shared libbusybox library and set of binary packages.</li>
	<li>Port opkg to work with more recent versions of BusyBox's shared library.</li>
	<li>Modify opkg to dynamically link against libbusybox.</li>
	<li>Add <code>Build-Depends</code> and <code>Depends</code> fields to the <code>opkg</code> source package.</li>
	<li>Investigate API and ABI stability of libbusybox.  This may require coordination with BusyBox developers.</li>
	<li>Look at each patch in opkg against libbb.  If it's still useful in newer versions, then refresh it, apply it to the <code>busybox</code> source package, and forward it upstream to BusyBox.<li>
</ul>
<!--#include virtual="../includes/footer.html" -->