summaryrefslogtreecommitdiffstats
path: root/dev/todo/buildd-infra.mdwn
blob: bba359cf446004b91883edda51b13592c0807bac (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
[[!meta title="Build Daemon Infrastructure"]]

Summary
=======

This pages documents the proposed infrastructure for handling package uploads,
builds, and archive inclusions.

Only source package uploads are accepted, and the process is simple and acyclic.
SFTP is used to upload packages for ease of maintainer authorization (uploaded
files need not be signed; uploaders are simply authorized by their SSH keypair).


Process
=======

The following diagram summarizes the process:

[[!img dev/buildd-infra.png size="x450" alt="buildd infrastructure diagram"]]

The general process is as follows:

 1. A maintainer uploads a source package (not necessarily his or her own –
    non-maintainer uploads are allowed) and changes file over SFTP into the
    review queue.  (This can be done by hand or with the assistance of a script
    similar to Debian's dupload and dput scripts.)
 2. A reviewer (whose job is similar to that of Debian's ftpmaster team members)
    examines the changes file and, if necessary, the source package itself and
    decides to either accept or reject the upload.  If accepted, the package is
    added to the master build queue and all of the relevant build queues.
 3. Each build daemon periodically checks the queues for which it is
    responsible.  If packages are waiting to be built, it attempts to build the
    oldest one (first in, first out).  If the build is successful, the daemon
    adds the produced binary packages to the archive queue.  Either way, the
    queue run ends with the removal of the processed source package from the
    relevant queue.
 4. When the master build daemon detects that a package no longer exists in any
    architecture build queue, it moves the source package and changes file into
    the archive queue.
 5. When the archive inclusion daemon finds a source package in the archive
    queue, it includes in the archive the source package along with all of the
    built binary packages.
 6. Users can finally download the source and binary packages from the archive
    using opkg.


Daemons and Utilities
=====================

buildq
------

**buildq** will be a utility used by upload reviewer(s) to either accept or
reject package uploads.

As this is written, uploads and build daemons must all be on the same
filesystem.  This can be changed someday by introducing a new queueing daemon
and reducing **buildq** to a simple uploader (removing all queueing logic).

This utility handles step 2 in the process above.

Usage:

    buildq accept <source> <version>
    buildq reject <source> <version>

Operation:

    write remarks template to file
    $EDITOR $remarks
    if action == accept
        touch src:$source_$version_src.opk
        ln src:$source_$version_src.opk $masterbuildqueue/
        remove " all " from space-padded Architecture field value
        if value changed
            ln src:$source_$version_src.opk $archindepbuildqueue/
        remove " any " from space-padded Architecture field value
        if value changed
            foreach archbuildqueue
                ln src:$source_$version_src.opk $archbuildqueue/
        else
            foreach archbuildqueue
                if Architecture value matches queue arch (use libopkh for this)
                    ln src:$source_$version_src.opk $archbuildqueue/
    rm src:$source_$version_src.opk
    mail maintainer and uploader

Example mail message:

    From: ProteanOS uploads team <proteanos-uploads@lists.proteanos.com>
    To: ProteanOS Foo Maintainers <proteanos-pkg-foo@lists.proteanos.com>,
     "P. J. McDermott" <pehjota@proteanos.com>
    Subject: Accepted foo 1.0-1
    
    Remarks:
     Your upload has been accepted.  The package will be serviced by the 
     build daemons shortly.  You will be notified if your package fails to
     build for any architectures or platforms.
    Source: foo
    Version: 1.0-1
    Architecture: any all
    Platform: all
    Distribution: trunk
    Maintainer: ProteanOS Foo Maintainers <proteanos-pkg-foo@lists.proteanos.com>
    Changed-By: "P. J. McDermott" <pehjota@proteanos.com>
    Date: Sat, 04 Aug 2012 10:31:40 -0400
    Changes:
     foo (1.0-1) trunk
     .
       * New package.

buildd
------

**buildd** will be a daemon that builds packages in its queue(s).

One **buildd** instance can be responsible for multiple architectures if the
queues are linked together.

This daemon handles step 3 in the process above.

Operation:

    while true
        headsrcopk = peek $queue (ls -1cr $queue | head -n 1)
        if no $headsrcopk
            sleep 00:05:00
            continue
        foreach host architecture+platform
            if ! Architecture value matches host arch (use libopkh for this)
                continue
            run builder on $headsrcopk
            if success
                mv *.opk $archivequeue/$source
            else
                rm *.opk
                touch $masterbuildqueue/$source_$version.fail
                mail log to maintainer and uploader
        pop $headsrcopk (rm $headsrcopk)

builddd
-------

A possible name expansion is "[builded][wikt-builded] it" (as in "Where's my
package? Have the build daemons builded it yet?") when spoken quickly.

**builddd** will be a daemon that watches the build daemons and acts on
successful and failed builds.

This daemon handles step 4 in the process above.

Operation:

    while true
        doneheadsrcopk = oldest opk in $masterbuildqueue with exactly 1 link
        if no $doneheadsrcopk
            sleep 00:05:00
            continue
        if no $masterbuildqueue/$source_$version.fail
            mv $doneheadsrcopk $archivequeue/$package_$version
        else
            rm -Rf $archivequeue/$package_$version

[wikt-builded]: https://en.wiktionary.org/wiki/builded

archived
--------

**archived** will be a daemon that includes built binary packages into the
package archive.

This daemon handles step 5 in the process above.

Operation:

    while true
        foreach dir in $archivequeue
            if exists src opk in $dir
                include $dir/*.opk

Filesystem Layout
-----------------

     +- <uploads-root>/
     |   \- incoming/
     +- <build-queue-root>/
     |   +- master/
     |   +- arch/
     |   |   +- all/ -> ../buildd/core/
     |   |   +- core-linux-eglibc/ -> ../buildd/core/
     |   |   +- i686-linux-eglibc/ -> ../buildd/i686/
     |   |   +- cortexa8-linux-eglibc/ -> ../buildd/cortexa8/
     |   |   \- cortexa8-linux-uclibc/ -> ../buildd/cortexa8/
     |   \- buildd/
     |       +- core/
     |       +- i686/
     |       \- cortexa8/
     \- <archive-queue-root>/
         \- <source>_<source-version>/