[[!meta title="Basic Packaging Tutorial"]] In this introduction to software packaging, we will package the Expat XML parser library. This is a pretty simple but complete package, consisting of a shared library and its development files plus an executable utility and some documentation. We will build a package in [Source Package Format 2.0][spf] (SPF 2.0) with the assistance of [[opkhelper_3.0.x|dev/opkhelper/]]. This tutorial assumes some knowledge of the UNIX shell command language and utilities (see the "Shell and Utilities" volume of [POSIX.1-2008][posix]) and at least basic familiarity with [makefile syntax][posix-makefile]. This tutorial presents one possible packaging workflow that seems to work well. There is no mandatory workflow to packaging. The only requirements are those made by the source package format and any build helper utilities that are used. [opkbuild]: http://git.proteanos.com/opkbuild/opkbuild.git/ [spf]: http://specs.proteanos.com/spf-2.0/ [opkhelper]: http://git.proteanos.com/opkhelper/opkhelper.git/ [posix]: http://pubs.opengroup.org/onlinepubs/9699919799/ [posix-makefile]: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/make.html#tag_20_76_13 Getting Started =============== Source Package Directory ------------------------ First, make a *source package directory*. This is the directory that will contain all of our source package files. SPF 2.0 makes no requirements on the name of this directory, but using the name of the source package is recommended. $ mkdir expat $ cd expat We need [a file called `format`][spf-format] to identify the format of our source package. For SPF 2.0, it should simply contain the string `2.0`. $ echo '2.0' >format Upstream Source Archive ----------------------- Obviously we need the source code of the software to be packaged. Go to [Expat's Web site][expat], find the expat 2.1.0 archive, and download it into the source package directory. $ wget 'http://downloads.sourceforge.net/project/expat/expat/2.1.0/expat-2.1.0.tar.gz' SPF 2.0 requires that an [upstream source archive][spf-upstream-source] be named `-.orig.tar`, where `` is the name of the source package, `` is the upstream version of the source package, and `` is an optional file extension to indicate compression. So, rename the archive accordingly. $ mv 'expat-2.1.0.tar.gz' 'expat-2.1.0.orig.tar.gz' [spf-format]: http://specs.proteanos.com/spf-2.0/overview.html#files-format [expat]: http://expat.sourceforge.net/ [spf-upstream-source]: http://specs.proteanos.com/spf-2.0/overview.html#files-src-src-ver-tar-ext Source Package Metadata ======================= Now we need some metadata for our source package. Control File ------------ First we'll make a `control` file. The format of this file is not yet documented in the SPF 2.0 specification, but it is documented [in the Debian Policy Manual][dpm-control]. The [source package fields][spf-fields-src] are `Maintainer` (required), `Build-Depends` (optional), and `Homepage` (optional). We'll fill in the fields whose values we know right now: `Maintainer` and `Homepage`. `Maintainer` is the name and e-mail address of the person or team responsible for the package (i.e. usually you when you are making a package). The value must follow the syntax of the `mailbox` symbol in [RFC 5322 section 3.4][rfc-5322-3.4]. That is, the value must be of the form `name
`. If `name` contains any of the following characters, it must be in double quotes: ( ) < > [ ] : ; @ \ , . `Homepage` is the URL of the Web site for the package, if such a site exists. Our expat `control` file looks like this: Maintainer: "J. Random Hacker" Homepage: http://expat.sourceforge.net/ Change Log ---------- Now we'll make a `changelog` file. The format of this file is documented [in the SPF 2.0 specification][spf-changelog]. We're making version "2.1.0-1" of the "expat" source package for the "trunk" distribution. We can get the current date and time in the RFC 5322 format using the **date**(1) command: $ LC_ALL='POSIX' date '+%a, %d %b %Y %H:%M:%S %z' Our expat `changelog` file looks like this: expat (2.1.0-1) trunk * Initial release. -- "J. Random Hacker" Sun, 18 Nov 2012 11:58:19 -0500 Building the Software ===================== We can now write our `build` makefile to try to get the Expat software to build. [The `build` makefile][spf-build] "directs the process of building and installing data files to be provided by binary packages". Looking Through the Source -------------------------- With a "[no-op][no-op]" target in `build`, we can make **opkbuild**(1) prepare a *[build work area][spf-work-area]* with the unpacked source code and stop. This target isn't required by SPF 2.0, but it seems to facilitate a nice workflow. So begin writing `build` as follows: #!/usr/bin/make -f nop: @: Note that, due to makefile syntax, the line after `nop:` must begin with a tab character. This line is called a "command line" in makefile syntax. The [`:` utility][posix-colon] is a "null utility" that returns an exit status of zero. A command prefix of `@` tells **make**(1) to not write the command to standard output before executing it. The `build` makefile must be executable, so set its file mode: $ chmod 755 build We can now make **opkbuild**(1) prepare our build work area. $ opkbuild -b -c -T nop The options are explained in the help output of opkbuild, obtained by running `opkbuild -h`. The `-b` option tells **opkbuild**(1) to build only binary packages (no source package). The `-c` option tells it to not clean up the work area after building packages. The `-T` option specifies a target to be built instead of the standard `build` and `install` targets. Now look in `tmp/src/`, the location of the source code within the build work area. $ ls tmp/src/ Look for some documentation file that might tell us how to build Expat. This kind of information is usually kept in a file called `INSTALL` or `README`. Expat's `README` file says to run `./configure`, then `make` and `make install`. Looking at `tmp/src/configure`, we see that it is "[g]enerated by GNU Autoconf 2.68 for expat 2.1.0". The `tmp/src/README` file reports that the makefile supports the use of either the `DESTDIR` or `INSTALL_ROOT` macro to install Expat somewhere other than in the root of the filesystem. So, we should be able to use opkhelper's buildsystem utilities to automatically configure, build, and install Expat for us. Building -------- So let's add a `build` target to our `build` makefile. The makefile should now look like this: #!/usr/bin/make -f nop: @: build: oh-autoconfigure oh-autobuild touch $@ Read the manual pages and/or source code of **oh-autoconfigure**(1) and **oh-autobuild**(1) to learn more about what they do. The `touch $@` command is recommended by SPF 2.0: > The build target should create a file named build in the build work area to > prevent configuration and compilation from being performed multiple times. We can now build Expat. $ opkbuild -b -c -T build [dpm-control]: http://www.debian.org/doc/debian-policy/ch-controlfields.html#s-controlsyntax [spf-fields-src]: http://specs.proteanos.com/spf-2.0/fields.html#fields-src [rfc-5322-3.4]: https://tools.ietf.org/html/rfc5322#section-3.4 [spf-changelog]: http://specs.proteanos.com/spf-2.0/metadata.html#changelog [spf-build]: http://specs.proteanos.com/spf-2.0/buildsys.html#build [no-op]: https://en.wiktionary.org/wiki/no-op [spf-work-area]: http://specs.proteanos.com/spf-2.0/buildsys.html#work-area [posix-colon]: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#colon Installing the Software ======================= We can now finish our `build` makefile to install the Expat software and make some binary packages. Installing ---------- Add a basic `install` target to the `build` makefile. The makefile should now look like this: #!/usr/bin/make -f nop: @: build: oh-autoconfigure oh-autobuild touch $@ install: build oh-autoinstall The `install` target is declared as depending on the `build` target: install: build Read the manual page and/or source code of **oh-autoinstall**(1) to learn more about what it does. Install Expat: $ opkbuild -b -c -T install Splitting Files Into Binary Packages ------------------------------------ Look in the *installation destination directory* `tmp/dest/` for files installed by Expat's build system. This can be done with the **find**(1) command, which results in the following when building for the `core-linux-eglibc` architecture: $ find tmp/dest -exec ls -Fd '{}' ';' | sed 's|^tmp/dest||' / /usr/ /usr/bin/ /usr/bin/xmlwf* /usr/share/ /usr/share/man/ /usr/share/man/man1/ /usr/share/man/man1/xmlwf.1 /usr/lib/ /usr/lib/core-linux-eglibc/ /usr/lib/core-linux-eglibc/pkgconfig/ /usr/lib/core-linux-eglibc/pkgconfig/expat.pc /usr/lib/core-linux-eglibc/libexpat.so@ /usr/lib/core-linux-eglibc/libexpat.a /usr/lib/core-linux-eglibc/libexpat.la* /usr/lib/core-linux-eglibc/libexpat.so.1@ /usr/lib/core-linux-eglibc/libexpat.so.1.6.0* /usr/include/ /usr/include/expat_external.h /usr/include/expat.h We have the `libexpat.so.1.6.0` shared library and two symbolic links to it: `libexpat.so.1` and `libexpat.so`. We have the `libexpat.a` static library and associated `libexpat.la` library metadata file generated by GNU libtool. We have a pkg-config file and two header files. We have an executable utility and an associated manual page. We should therefore split these files into four binary packages: one for the shared library, one for the library development files, one for the utility, and one for the utility's documentation. To find out what we should call the library package, we can use **objdump**(1) to get the *SONAME* of the library: $ objdump -p tmp/dest/usr/lib/core-linux-eglibc/libexpat.so.1.6.0 | grep SONAME SONAME libexpat.so.1 We should name our library package after the SONAME of the shared library, without `.so`. The binary package shall be named **`libexpat.1`**. The versionless `libexpat.so` link is only needed by **ld**(1) when linking a just-compiled object with the `-lexpat` linker flag. So this can be provided by our library development package. Also provided by that package will be the header files, the pkg-config file, and the static library. The development package can be called **`libexpat.1-dev`**. The `xmlwf` utility can be provided by a package called simply **`xmlwf`**. The `xmlwf.1` manual page can be provided by a package called **`xmlwf-doc`**. Binary Packages =============== Binary Package Metadata ----------------------- Each binary package to be built needs to have [a directory for its metadata][spf-binpkg.pkg]. So let's create directories for our packages. $ mkdir libexpat.1.pkg libexpat.1-dev.pkg xmlwf.pkg xmlwf-doc.pkg SPF 2.0 requires a `control` file for each binary package. The format of this file is the same as that of the source package `control` file. The required [binary package fields][spf-fields-bin] are `Architecture`, `Platform`, and `Description`. None of these binary packages are platform-specific, so they will all have a `Platform: all` field. All of the binary packages except `xmlwf-doc` are architecture-specific; that is, they provide files whose contents depend on the host architecture (files like executable and linkable objects). So `xmlwf-doc` will have an `Architecture: all` field while the others will have `Architecture: any` fields. Let's start with the `libexpat.1.pkg/control` file: Architecture: any Platform: all Description: XML parser library Expat is an XML parser library written in C. It is a stream-oriented parser in which an application registers handlers for things the parser might find in the XML document (like start tags). That's fairly simple. Now let's write a `control` file for `libexpat.1-dev`. Because it provides development files for `libexpat.so.1`, `libexpat.1-dev` should depend on the `libexpat.1` package. This should be a versioned dependency, because the `libexpat.so` symbolic link points to a specific version of `libexpat.so`. Architecture: any Platform: all Depends: libexpat.1 (= 2.1.0-1) Description: XML parser library - development files Expat is an XML parser library written in C. It is a stream-oriented parser in which an application registers handlers for things the parser might find in the XML document (like start tags). . This package provides development files for Expat. Next is `xmlwf`, which should also depend on `libexpat.1` since the `xmlwf` utility is dynamically linked against the `libexpat.so.1` library. Architecture: any Platform: all Depends: libexpat.1 Description: XML parser library - example application This package provides an example application of Expat that determines if an XML document is well-formed. Finally, we can write metadata for `xmlwf-doc`, which should depend on `xmlwf` since it documents the `xmlwf` utility. Architecture: all Platform: all Depends: xmlwf Description: XML parser library - example application documentation files This package provides the manual page for xmlwf, an example application of Expat that determines if an XML document is well-formed. Binary Package Data Files ------------------------- The **oh-installfiles**(1) utility of opkhelper, which we'll be using to install files into *binary package data directories*, requires a `files` file for each binary package that is to provide data files. Recall how we decided to split files between packages. We will now write pathname patterns to do this. Again, let's start with `libexpat.1`. We can write the following pattern in `libexpat.1.pkg/files`: /usr/lib/*/libexpat.so.* This will match `/usr/lib/core-linux-eglibc/libexpat.so.1` and `/usr/lib/core-linux-eglibc/libexpat.so.1.6.0`; these two files will be provided by `libexpat.1`. The patterns for `libexpat.1-dev` are a little more complicated: /usr/include /usr/lib/*/libexpat.so /usr/lib/*/libexpat.a /usr/lib/*/pkgconfig The first pattern simply matches the directory containing header files. The second matches the versionless symbolic link; remember this is used by **ld**(1) to link a just-compiled object against `libexpat.so.1.6.0`. The third matches the static library, and the fourth matches the directory containing the `expat.pc` pkg-config file. `xmlwf.pkg/files` need only contain a pattern to match the directory containing the `xmlwf` utility. /usr/bin `xmlwf-doc.pkg/files` is similarly simple: /usr/share/man/man1 With these pathname patterns done, we can add **oh-installfiles**(1) to our `build` makefile: #!/usr/bin/make -f nop: @: build: oh-autoconfigure oh-autobuild touch $@ install: build oh-autoinstall oh-installfiles Now run **opkbuild**(1) again: $ opkbuild -b -c -T install You can verify that all files were installed where they should be: $ find tmp/*.data -exec ls -Fd '{}' ';' tmp/libexpat.1.data/ tmp/libexpat.1.data/usr/ tmp/libexpat.1.data/usr/lib/ tmp/libexpat.1.data/usr/lib/core-linux-eglibc/ tmp/libexpat.1.data/usr/lib/core-linux-eglibc/libexpat.so.1@ tmp/libexpat.1.data/usr/lib/core-linux-eglibc/libexpat.so.1.6.0* tmp/libexpat.1-dev.data/ tmp/libexpat.1-dev.data/usr/ tmp/libexpat.1-dev.data/usr/lib/ tmp/libexpat.1-dev.data/usr/lib/core-linux-eglibc/ tmp/libexpat.1-dev.data/usr/lib/core-linux-eglibc/pkgconfig/ tmp/libexpat.1-dev.data/usr/lib/core-linux-eglibc/pkgconfig/expat.pc tmp/libexpat.1-dev.data/usr/lib/core-linux-eglibc/libexpat.so@ tmp/libexpat.1-dev.data/usr/lib/core-linux-eglibc/libexpat.a tmp/libexpat.1-dev.data/usr/include/ tmp/libexpat.1-dev.data/usr/include/expat_external.h tmp/libexpat.1-dev.data/usr/include/expat.h tmp/xmlwf.data/ tmp/xmlwf.data/usr/ tmp/xmlwf.data/usr/bin/ tmp/xmlwf.data/usr/bin/xmlwf* tmp/xmlwf-doc.data/ tmp/xmlwf-doc.data/usr/ tmp/xmlwf-doc.data/usr/share/ tmp/xmlwf-doc.data/usr/share/man/ tmp/xmlwf-doc.data/usr/share/man/man1/ tmp/xmlwf-doc.data/usr/share/man/man1/xmlwf.1 Cleaning Up Installed Files --------------------------- There are few things we can do to improve our `build` makefile's `install` target. You may have noticed **oh-installfiles**(1) warn that something hasn't been installed: > oh-installfiles: Warning: Some files have not been installed into packages With **find**(1), we can see that this is the `libexpat.la` file that GNU libtool generated. $ find tmp/dest -type f -exec ls -Fd '{}' ';' | sed 's|^tmp/dest||' /usr/lib/core-linux-eglibc/libexpat.la* We don't need this, and we can simply delete it in the `install` target. Next, note that some file permissions aren't entirely correct. For example, `libexpat.so.1.6.0` is executable, but almost all libraries need not be. So we can call **oh-fixperms**(1) in our `install` target to automatically set correct permissions for us. Finally, note that the executable and linkable objects are not stripped: they contain all of their symbols, including those only needed for debugging. $ file tmp/libexpat.1.data/usr/lib/core-linux-eglibc/libexpat.so.1.6.0 tmp/libexpat.1.data/usr/lib/core-linux-eglibc/libexpat.so.1.6.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0x2d88e36feeb8245bfa2f63f2f0e9a9f8232f6d2c, not stripped $ file tmp/xmlwf.data/usr/bin/xmlwf tmp/xmlwf.data/usr/bin/xmlwf: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.26, BuildID[sha1]=0xdb5f686930b13b8a5e7519efb446a2da14de9856, not stripped We can call **oh-strip**(1) in our `install` target to automatically strip objects for us. So our `build` makefile should now look like this: #!/usr/bin/make -f nop: @: build: oh-autoconfigure oh-autobuild touch $@ install: build oh-autoinstall rm -f 'dest/usr/lib/$(OPK_HOST_ARCH)/libexpat.la' oh-fixperms oh-strip oh-installfiles [spf-binpkg.pkg]: http://specs.proteanos.com/spf-2.0/overview.html#files-binpkg.pkg [spf-fields-bin]: http://specs.proteanos.com/spf-2.0/fields.html#fields-src Documentation and Finishing Touches =================================== Source Package Documentation ---------------------------- SPF 2.0 [specifies][spf-docs] that one of the binary packages built from a source package provides documentation files about the source package and is depended upon by all of the other binary packages from the source package. So we should pick one common binary package that should be a dependency of all of our other binary packages. `libexpat.1` is a good candidate for this, since it is already a direct dependency of `libexpat.1-dev` and `xmlwf` and an indirect dependency of `xmlwf-doc`. Per SPF 2.0, we can mark `libexpat.1` as providing source package documentation by making a `docs` file in its metadata directory. $ touch libexpat.1.pkg/docs We should make all of our other binary packages directly depend on `libexpat.1` version `2.1.0-1`. For example, `xmlwf-doc.pkg/control` should now look like this: Architecture: all Platform: all Depends: libexpat.1 (= 2.1.0-1), xmlwf Description: XML parser library - example application documentation files This package provides the manual page for xmlwf, an example application of Expat that determines if an XML document is well-formed. Substitution Variables ---------------------- We've hardcoded the `libexpat.1` binary package version in many of our control files. What will we do when we make a new version of our source package? We'll have to change all of these values in all of these places. [*Substitution variables*][spf-substvars] (*substvars* for short) make this unnecessary. We can just use the `Binary-Version` substitution variable in our control files to refer to the version of our binary packages. For example, our `xmlwf-doc.pkg/control` file should now look like this: Architecture: all Platform: all Depends: libexpat.1 (= ${Binary-Version}), xmlwf Description: XML parser library - example application documentation files This package provides the manual page for xmlwf, an example application of Expat that determines if an XML document is well-formed. But that's not all! We can define our own variables as well. Note that the descriptions of our `libexpat.1` and `libexpat.1-dev` packages have a common paragraph. We can put that in a file called `substvars`: Common-Description: Expat is an XML parser library written in C. It is a stream-oriented parser in which an application registers handlers for things the parser might find in the XML document (like start tags). As noted by the SPF 2.0 specification, the leading newline character in the value is fine: > Values may be comprised of multiple lines, and empty lines at the beginning > and end of each substitution variable value shall be removed. We can now use this variable in our `control` files. Here's `libexpat.1.pkg/control`: Architecture: any Platform: all Description: XML parser library ${Common-Description} And here's `libexpat.1-dev.pkg/control`: Architecture: any Platform: all Depends: libexpat.1 (= ${Binary-Version}) Description: XML parser library - development files ${Common-Description} . This package provides development files for Expat. Copyright and License Information --------------------------------- We're almost done; we just have one more important thing to do. We need to document the copyright information for the upstream software and our own packaging work. This is done in the `copyright` file. There is currently no standard format for this file. We need to collect the copyright and license information from the upstream source code (usually in comments at the tops of source files). There are some resources available to assist us with this. First, we can look at the work already done by package maintainers in the Debian Project. Find the [copyright file][deb-expat-copyright] for Debian's `expat` source package. We see the following copyright information: Copyright (c) 1998, 1999, 2000 Thai Open Source Software Center Ltd and Clark Cooper Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006 Expat maintainers. We also see that Expat can be dealt in under the terms of, unsurprisingly, the Expat (a.k.a. "MIT") license. Another resource we can use is the [**licensecheck**(1) tool][licensecheck], maintained in Debian's `devscripts` package and originally based on a script from the KDE SDK. Recursively run **licensecheck**(1) to report copyright and license information. $ licensecheck -r --copyright tmp/src/ We see that some source files have publication dates in their copyright notices that are newer than those that Debian's copyright file lists: tmp/src/amiga/expat_lib.c: MIT/X11 (BSD like) [Copyright: 2001-2009 Expat maintainers / HOLDERS BE LIABLE FOR ANY] So collect some representative copyright notices – e.g. from `tmp/src/lib/xmlparse.c`, `tmp/src/examples/outline.c`, `tmp/src/vms/expat_config.h`, and `tmp/src/amiga/expat_lib.c` – and add them to the `copyright` file. Then describe the license under which the software may be used. `Expat` is a "common license" included under `/usr/share/common-licenses/` in this distribution, so you can refer to it there. You should also document the location from which the source was obtained. Finally, add your own copyright notice and license information. You should allow your work to be used under the terms of a license that is equivalent to or compatible with the terms of the upstream software's copyright license. Your resulting `copyright` file might look something like this: Upstream Source =============== Location: Copyright (c) 1998, 1999, 2000 Thai Open Source Software Center Ltd Copyright 1999, Clark Cooper Copyright 2000, Clark Cooper Copyright (c) 2001-2009 Expat maintainers. These files may be reproduced, distributed, modified, and otherwise dealt in under the terms of the Expat License. On this system, a copy of the Expat License may be found at . Distribution Packaging ====================== Copyright (C) 2012 J. Random Hacker These files may be reproduced, distributed, modified, and otherwise dealt in under the terms of the Expat License. On this system, a copy of the Expat License may be found at . Building Everything ------------------- Now we can build all of our source and binary packages and verify that everything is correct. **opkbuild**(1) maintains a cache file in the work area; because we've modified the metadata in our packaging since the first time we ran **opkbuild**(1), this cache file is out-of-date. Also, we should make sure that the entire build process still works. So let's clean up the work area before going any further. $ rm -Rf tmp/ Now let's run **opkbuild**(1) again, this time completely building all of our source and binary packages and cleaning up automatically when we're done. $ opkbuild After that finishes, you should see the built packages in the parent directory. $ ls -1 ../*.opk ../libexpat.1_2.1.0-2_core-linux-eglibc_all.opk ../libexpat.1-dev_2.1.0-2_core-linux-eglibc_all.opk ../src-expat_2.1.0-2_src_all.opk ../xmlwf_2.1.0-2_core-linux-eglibc_all.opk ../xmlwf-doc_2.1.0-2_all_all.opk `src-expat` is a *source binary package* – a binary package installable with the package manager that provides the files in our source package. This binary package is a convenient way to distribute our source package to others. You can use the **tar**(1) command to verify that the control information and data files in packages look correct. $ tar -xzO control.tar.gz \ > <../libexpat.1_2.1.0-2_core-linux-eglibc_all.opk | tar -xzO ./control Package: libexpat.1 Source: expat Version: 2.1.0-2 Architecture: core-linux-eglibc Platform: all Maintainer: "J. Random Hacker" Installed-Size: 164 Description: XML parser library Expat is an XML parser library written in C. It is a stream-oriented parser in which an application registers handlers for things the parser might find in the XML document (like start tags). Homepage: http://expat.sourceforge.net/ $ tar -xzO data.tar.gz \ > <../libexpat.1_2.1.0-2_core-linux-eglibc_all.opk | tar -tz ./ ./usr/ ./usr/share/ ./usr/share/doc/ ./usr/share/doc/libexpat.1/ ./usr/share/doc/libexpat.1/changelog.dist ./usr/share/doc/libexpat.1/copyright ./usr/lib/ ./usr/lib/core-linux-eglibc/ ./usr/lib/core-linux-eglibc/libexpat.so.1 ./usr/lib/core-linux-eglibc/libexpat.so.1.6.0 Congratulations! You've made a source package that successfully builds four binary packages! [spf-docs]: http://specs.proteanos.com/spf-2.0/metadata.html#docs [spf-substvars]: http://specs.proteanos.com/spf-2.0/substvars.html [deb-expat-copyright]: http://packages.debian.org/changelogs/pool/main/e/expat/current/copyright [licensecheck]: http://anonscm.debian.org/gitweb/?p=devscripts/devscripts.git;a=blob;f=scripts/licensecheck.pl;hb=HEAD