[[!meta title="Build Daemon Infrastructure"]] Summary ======= This pages documents the proposed infrastructure for handling package uploads, builds, and archive inclusions. Only source package uploads are accepted, and the process is simple and acyclic. SFTP is used to upload packages for ease of maintainer authorization (uploaded files need not be signed; uploaders are simply authorized by their SSH keypair). Process ======= The following diagram summarizes the process: [[!img dev/buildd-infra.png size="x450" alt="buildd infrastructure diagram"]] The general process is as follows: 1. A maintainer uploads a source package (not necessarily his or her own – non-maintainer uploads are allowed) and changes file over SFTP into the review queue. (This can be done by hand or with the assistance of a script similar to Debian's dupload and dput scripts.) 2. A reviewer (whose job is similar to that of Debian's ftpmaster team members) examines the changes file and, if necessary, the source package itself and decides to either accept or reject the upload. If accepted, the package is added to the master build queue and all of the relevant build queues. 3. Each build daemon periodically checks the queues for which it is responsible. If packages are waiting to be built, it attempts to build the oldest one (first in, first out). If the build is successful, the daemon adds the produced binary packages to the archive queue. Either way, the queue run ends with the removal of the processed source package from the relevant queue. 4. When the master build daemon detects that a package no longer exists in any architecture build queue, it moves the source package and changes file into the archive queue. 5. When the archive inclusion daemon finds a source package in the archive queue, it includes in the archive the source package along with all of the built binary packages. 6. Users can finally download the source and binary packages from the archive using opkg. Daemons and Utilities ===================== buildq ------ **buildq** will be a utility used by upload reviewer(s) to either accept or reject package uploads. As this is written, uploads and build daemons must all be on the same filesystem. This can be changed someday by introducing a new queueing daemon and reducing **buildq** to a simple uploader (removing all queueing logic). This utility handles step 2 in the process above. Usage: buildq accept buildq reject Operation: write remarks template to file $EDITOR $remarks if action == accept touch src:$source_$version_src.opk ln src:$source_$version_src.opk $masterbuildqueue/ remove " all " from space-padded Architecture field value if value changed ln src:$source_$version_src.opk $archindepbuildqueue/ remove " any " from space-padded Architecture field value if value changed foreach archbuildqueue ln src:$source_$version_src.opk $archbuildqueue/ else foreach archbuildqueue if Architecture value matches queue arch (use libopkh for this) ln src:$source_$version_src.opk $archbuildqueue/ rm src:$source_$version_src.opk mail maintainer and uploader Example mail message: From: ProteanOS uploads team To: ProteanOS Foo Maintainers , "P. J. McDermott" Subject: Accepted foo 1.0-1 Remarks: Your upload has been accepted. The package will be serviced by the build daemons shortly. You will be notified if your package fails to build for any architectures or platforms. Source: foo Version: 1.0-1 Architecture: any all Platform: all Distribution: trunk Maintainer: ProteanOS Foo Maintainers Changed-By: "P. J. McDermott" Date: Sat, 04 Aug 2012 10:31:40 -0400 Changes: foo (1.0-1) trunk . * New package. buildd ------ **buildd** will be a daemon that builds packages in its queue(s). One **buildd** instance can be responsible for multiple architectures if the queues are linked together. This daemon handles step 3 in the process above. Operation: while true headsrcopk = peek $queue (ls -1cr $queue | head -n 1) if no $headsrcopk sleep 00:05:00 continue foreach host architecture+platform if ! Architecture value matches host arch (use libopkh for this) continue run builder on $headsrcopk if success mv *.opk $archivequeue/$source else rm *.opk touch $masterbuildqueue/$source_$version.fail mail log to maintainer and uploader pop $headsrcopk (rm $headsrcopk) builddd ------- A possible name expansion is "[builded][wikt-builded] it" (as in "Where's my package? Have the build daemons builded it yet?") when spoken quickly. **builddd** will be a daemon that watches the build daemons and acts on successful and failed builds. This daemon handles step 4 in the process above. Operation: while true doneheadsrcopk = oldest opk in $masterbuildqueue with exactly 1 link if no $doneheadsrcopk sleep 00:05:00 continue if no $masterbuildqueue/$source_$version.fail mv $doneheadsrcopk $archivequeue/$package_$version else rm -Rf $archivequeue/$package_$version [wikt-builded]: https://en.wiktionary.org/wiki/builded archived -------- **archived** will be a daemon that includes built binary packages into the package archive. This daemon handles step 5 in the process above. Operation: while true foreach dir in $archivequeue if exists src opk in $dir include $dir/*.opk Filesystem Layout ----------------- +- / | \- incoming/ +- / | +- master/ | +- arch/ | | +- all/ -> ../buildd/core/ | | +- core-linux-eglibc/ -> ../buildd/core/ | | +- i686-linux-eglibc/ -> ../buildd/i686/ | | +- cortexa8-linux-eglibc/ -> ../buildd/cortexa8/ | | \- cortexa8-linux-uclibc/ -> ../buildd/cortexa8/ | \- buildd/ | +- core/ | +- i686/ | \- cortexa8/ \- / \- _/