1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
|
[[!meta title="Build Daemon Infrastructure"]]
Summary
=======
This pages documents the proposed infrastructure for handling package uploads,
builds, and archive inclusions.
Only source package uploads are accepted, and the process is simple and acyclic.
SFTP is used to upload packages for ease of maintainer authorization (uploaded
files need not be signed; uploaders are simply authorized by their SSH keypair).
Process
=======
The following diagram summarizes the process:
[[!img dev/buildd-infra.png size="x450" alt="buildd infrastructure diagram"]]
The general process is as follows:
1. A maintainer uploads a source package (not necessarily his or her own –
non-maintainer uploads are allowed) and changes file over SFTP into the
review queue. (This can be done by hand or with the assistance of a script
similar to Debian's dupload and dput scripts.)
2. A reviewer (whose job is similar to that of Debian's ftpmaster team members)
examines the changes file and, if necessary, the source package itself and
decides to either accept or reject the upload. If accepted, the package is
added to the master build queue and all of the relevant build queues.
3. Each build daemon periodically checks the queues for which it is
responsible. If packages are waiting to be built, it attempts to build the
oldest one (first in, first out). If the build is successful, the daemon
adds the produced binary packages to the archive queue. Either way, the
queue run ends with the removal of the processed source package from the
relevant queue.
4. When the master build daemon detects that a package no longer exists in any
architecture build queue, it moves the source package and changes file into
the archive queue.
5. When the archive inclusion daemon finds a source package in the archive
queue, it includes in the archive the source package along with all of the
built binary packages.
6. Users can finally download the source and binary packages from the archive
using opkg.
Daemons and Utilities
=====================
buildq
------
**buildq** will be a utility used by upload reviewer(s) to either accept or
reject package uploads.
As this is written, uploads and build daemons must all be on the same
filesystem. This can be changed someday by introducing a new queueing daemon
and reducing **buildq** to a simple uploader (removing all queueing logic).
This utility handles step 2 in the process above.
Usage:
buildq accept <source> <version>
buildq reject <source> <version>
Operation:
write remarks template to file
$EDITOR $remarks
if action == accept
touch src:$source_$version_src.opk
ln src:$source_$version_src.opk $masterbuildqueue/
remove " all " from space-padded Architecture field value
if value changed
ln src:$source_$version_src.opk $archindepbuildqueue/
remove " any " from space-padded Architecture field value
if value changed
foreach archbuildqueue
ln src:$source_$version_src.opk $archbuildqueue/
else
foreach archbuildqueue
if Architecture value matches queue arch (use libopkh for this)
ln src:$source_$version_src.opk $archbuildqueue/
rm src:$source_$version_src.opk
mail maintainer and uploader
Example mail message:
From: ProteanOS uploads team <proteanos-uploads@lists.proteanos.com>
To: ProteanOS Foo Maintainers <proteanos-pkg-foo@lists.proteanos.com>,
"P. J. McDermott" <pehjota@proteanos.com>
Subject: Accepted foo 1.0-1
Remarks:
Your upload has been accepted. The package will be serviced by the
build daemons shortly. You will be notified if your package fails to
build for any architectures or platforms.
Source: foo
Version: 1.0-1
Architecture: any all
Platform: all
Distribution: trunk
Maintainer: ProteanOS Foo Maintainers <proteanos-pkg-foo@lists.proteanos.com>
Changed-By: "P. J. McDermott" <pehjota@proteanos.com>
Date: Sat, 04 Aug 2012 10:31:40 -0400
Changes:
foo (1.0-1) trunk
.
* New package.
buildd
------
**buildd** will be a daemon that builds packages in its queue(s).
One **buildd** instance can be responsible for multiple architectures if the
queues are linked together.
This daemon handles step 3 in the process above.
Operation:
while true
headsrcopk = peek $queue (ls -1cr $queue | head -n 1)
if no $headsrcopk
sleep 00:05:00
continue
foreach host architecture+platform
if ! Architecture value matches host arch (use libopkh for this)
continue
run builder on $headsrcopk
if success
mv *.opk $archivequeue/$source
else
rm *.opk
touch $masterbuildqueue/$source_$version.fail
mail log to maintainer and uploader
pop $headsrcopk (rm $headsrcopk)
builddd
-------
A possible name expansion is "[builded][wikt-builded] it" (as in "Where's my
package? Have the build daemons builded it yet?") when spoken quickly.
**builddd** will be a daemon that watches the build daemons and acts on
successful and failed builds.
This daemon handles step 4 in the process above.
Operation:
while true
doneheadsrcopk = oldest opk in $masterbuildqueue with exactly 1 link
if no $doneheadsrcopk
sleep 00:05:00
continue
if no $masterbuildqueue/$source_$version.fail
mv $doneheadsrcopk $archivequeue/$package_$version
else
rm -Rf $archivequeue/$package_$version
[wikt-builded]: https://en.wiktionary.org/wiki/builded
archived
--------
**archived** will be a daemon that includes built binary packages into the
package archive.
This daemon handles step 5 in the process above.
Operation:
while true
foreach dir in $archivequeue
if exists src opk in $dir
include $dir/*.opk
Filesystem Layout
-----------------
+- <uploads-root>/
| \- incoming/
+- <build-queue-root>/
| +- master/
| +- arch/
| | +- all/ -> ../buildd/core/
| | +- core-linux-eglibc/ -> ../buildd/core/
| | +- i686-linux-eglibc/ -> ../buildd/i686/
| | +- cortexa8-linux-eglibc/ -> ../buildd/cortexa8/
| | \- cortexa8-linux-uclibc/ -> ../buildd/cortexa8/
| \- buildd/
| +- core/
| +- i686/
| \- cortexa8/
\- <archive-queue-root>/
\- <source>_<source-version>/
|
