[[!meta title="Basic Packaging Tutorial"]] In this introduction to software packaging, we will package the Expat XML parser library. This is a pretty simple but complete package, consisting of a shared library and its development files plus an executable utility and some documentation. We will use [opkbuild 3.0.x][opkbuild] to build a package in [Source Package Format 2.0][spf] (SPF 2.0) with the assistance of [opkhelper 3.0.x][opkhelper]. This tutorial assumes some knowledge of the UNIX shell command language and utilities (see the "Shell and Utilities" volume of [POSIX.1-2008][posix]) and at least basic familiarity with [makefile syntax][posix-makefile]. This tutorial presents one possible packaging workflow that seems to work well. There is no mandatory workflow to packaging. The only requirements are those made by the source package format and any build helper utilities that are used. Getting Started =============== Source Package Directory ------------------------ First, make a *source package directory*. This is the directory that will contain all of our source package files. SPF 2.0 makes no requirements on the name of this directory, but using the name of the source package is recommended. $ mkdir expat $ cd expat We need [a file called `format`][spf-format] to identify the format of our source package. For SPF 2.0, it should simply contain the string `2.0`. $ echo '2.0' >format Upstream Source Archive ----------------------- Obviously we need the source code of the software to be packaged. Go to [Expat's Web site][expat], find the expat 2.1.0 archive, and download it into the source package directory. $ wget 'http://downloads.sourceforge.net/project/expat/expat/2.1.0/expat-2.1.0.tar.gz' SPF 2.0 requires that an [upstream source archive][spf-upstream-source] be named `-.orig.tar`, where `` is the name of the source package, `` is the upstream version of the source package, and `` is an optional file extension to indicate compression. So, rename the archive accordingly. $ mv 'expat-2.1.0.tar.gz' 'expat-2.1.0.orig.tar.gz' Source Package Metadata ======================= Now we need some metadata for our source package. Control File ------------ First we'll make a `control` file. The format of this file is not yet documented in the SPF 2.0 specification, but it is documented [in the Debian Policy Manual][dpm-control]. The [source package fields][spf-fields-src] are `Maintainer` (required), `Build-Depends` (optional), and `Homepage` (optional). We'll fill in the fields whose values we know right now: `Maintainer` and `Homepage`. `Maintainer` is the name and e-mail address of the person or team responsible for the package (i.e. usually you when you are making a package). The value must follow the syntax of the `mailbox` symbol in [RFC 5322 section 3.4][rfc-5322-3.4]. That is, the value must be of the form `name
`. If `name` contains any of the following characters, it must be in double quotes: ( ) < > [ ] : ; @ \ , . `Homepage` is the URL of the Web site for the package, if such a site exists. Our expat `control` file looks like this: Maintainer: "J. Random Hacker" Homepage: http://expat.sourceforge.net/ Change Log ---------- Now we'll make a `changelog` file. The format of this file is documented [in the SPF 2.0 specification][spf-changelog]. We're making version "2.1.0-1" of the "expat" source package for the "trunk" distribution. We can get the current date and time in the RFC 5322 format using the **date**(1) command: $ LC_ALL='POSIX' date '+%a, %d %b %Y %H:%M:%S %z' Our expat `changelog` file looks like this: expat (2.1.0-1) trunk * Initial release. -- "J. Random Hacker" Sun, 18 Nov 2012 11:58:19 -0500 Building the Software ===================== We can now write our `build` makefile to try to get the Expat software to build. [The `build` makefile][spf-build] "directs the process of building and installing data files to be provided by binary packages". Looking Through the Source -------------------------- With a "[no-op][no-op]" target in `build`, we can make **opkbuild**(1) prepare a *[build work area][spf-work-area]* with the unpacked source code and stop. This target isn't required by SPF 2.0, but it seems to facilitate a nice workflow. So begin writing `build` as follows: #!/usr/bin/make -f nop: @: Note that, due to makefile syntax, the line after `nop:` must begin with a tab character. This line is called a "command line" in makefile syntax. The [`:` utility][posix-colon] is a "null utility" that returns an exit status of zero. A command prefix of `@` tells make to not write the command to standard output before executing it. The `build` makefile must be executable, so set its file mode: $ chmod 755 build We can now make **opkbuild**(1) prepare our build work area. $ opkbuild -b -c -T nop The options are explained in the help output of opkbuild, obtained by running `opkbuild -h`. The `-b` option tells **opkbuild**(1) to build only binary packages (no source package). The `-c` option tells it to not clean up the work area after building packages. The `-T` option specifies a target to be built instead of the standard `build` and `install` targets. Now look in `tmp/src/`, the location of the source code within the build work area. $ ls tmp/src/ Look for some documentation file that might tell us how to build Expat. This kind of information is usually kept in a file called `INSTALL` or `README`. Expat's `README` file says to run `./configure`, then `make` and `make install`. Looking at `tmp/src/configure`, we see that it is "[g]enerated by GNU Autoconf 2.68 for expat 2.1.0". The `tmp/src/README` file reports that the makefile supports the use of either the `DESTDIR` or `INSTALL_ROOT` macro to install Expat somewhere other than in the root of the filesystem. So, we should be able to use opkhelper's buildsystem utilities to automatically configure, build, and install Expat for us. Building -------- So let's add a `build` target to our `build` makefile. The makefile should now look like this: #!/usr/bin/make -f nop: @: build: oh-autoconfigure oh-autobuild touch $@ Read the manual pages and/or source code of **oh-autoconfigure**(1) and **oh-autobuild**(1) to learn more about what they do. The `touch $@` command is recommended by SPF 2.0: > The build target should create a file named build in the build work area to > prevent configuration and compilation from being performed multiple times. We can now build Expat. $ opkbuild -b -c -T build Installing the Software ======================= We can now finish our `build` makefile to install the Expat software and make some binary packages. Installing ---------- Add a basic `install` target to the `build` makefile. The makefile should now look like this: #!/usr/bin/make -f nop: @: build: oh-autoconfigure oh-autobuild touch $@ install: build oh-autoinstall The `install` target is declared as depending on the `build` target: install: build Read the manual page and/or source code of **oh-autoinstall**(1) to learn more about what it does. Install Expat: $ opkbuild -b -c -T install Splitting Files Into Binary Packages ------------------------------------ Look in the *installation destination directory* `tmp/dest/` for files installed by Expat's build system. This can be done with the **find**(1) command, which results in the following when building for the `core-linux-eglibc` architecture: $ find tmp/dest -exec ls -Fd '{}' ';' | sed 's|^tmp/dest||' / /usr/ /usr/bin/ /usr/bin/xmlwf* /usr/share/ /usr/share/man/ /usr/share/man/man1/ /usr/share/man/man1/xmlwf.1 /usr/lib/ /usr/lib/core-linux-eglibc/ /usr/lib/core-linux-eglibc/pkgconfig/ /usr/lib/core-linux-eglibc/pkgconfig/expat.pc /usr/lib/core-linux-eglibc/libexpat.so@ /usr/lib/core-linux-eglibc/libexpat.a /usr/lib/core-linux-eglibc/libexpat.la* /usr/lib/core-linux-eglibc/libexpat.so.1@ /usr/lib/core-linux-eglibc/libexpat.so.1.6.0* /usr/include/ /usr/include/expat_external.h /usr/include/expat.h We have the `libexpat.so.1.6.0` shared library and two symbolic links to it: `libexpat.so.1` and `libexpat.so`. We have the `libexpat.a` static library and associated `libexpat.la` library metadata file generated by GNU libtool. We have a pkg-config file and two header files. We have an executable utility and an associated manual page. We should therefore split these files into four binary packages: one for the shared library, one for the library development files, one for the utility, and one for the utility's documentation. To find out what we should call the library package, we can use **objdump**(1) to get the *SONAME* of the library: $ objdump -p tmp/dest/usr/lib/core-linux-eglibc/libexpat.so.1.6.0 | grep SONAME SONAME libexpat.so.1 We should name our library package after the SONAME of the shared library, without `.so`. The binary package shall be named **`libexpat.1`**. The versionless `libexpat.so` link is only needed by **ld**(1) when linking a just-compiled object with the `-lexpat` linker flag. So this can be provided by our library development package. Also provided by that package will be the header files, the pkg-config file, and the static library. The development package can be called **`libexpat.1-dev`**. The `xmlwf` utility can be provided by a package called simply **`xmlwf`**. The `xmlwf.1` manual page can be provided by a package called **`xmlwf-doc`**. Binary Package Metadata ----------------------- Each binary package to be built needs to have [a directory for its metadata][spf-binpkg.pkg]. So let's create directories for our packages. $ mkdir libexpat.1.pkg libexpat.1-dev.pkg xmlwf.pkg xmlwf-doc.pkg SPF 2.0 requires a `control` file for each binary package. The format of this file is the same as that of the source package `control` file. The required [binary package fields][spf-fields-bin] are `Architecture`, `Platform`, and `Description`. None of these binary packages are platform-specific, so they will all have a `Platform: all` field. All of the binary packages except `xmlwf-doc` are architecture-specific; that is, they provide files whose contents depend on the host architecture (files like executable and linkable objects). So `xmlwf-doc` will have an `Architecture: all` field while the others will have `Architecture: any` fields. Let's start with the `libexpat.1.pkg/control` file: Architecture: any Platform: all Description: XML parser library Expat is an XML parser library written in C. It is a stream-oriented parser in which an application registers handlers for things the parser might find in the XML document (like start tags). That's fairly simple. Now let's write a `control` file for `libexpat.1-dev`. Because it provides development files for `libexpat.so.1`, `libexpat.1-dev` should depend on the `libexpat.1` package. This should be a versioned dependency, because the `libexpat.so` symbolic link points to a specific version of `libexpat.so`. Architecture: any Platform: all Depends: libexpat.1 (= 2.1.0-1) Description: XML parser library - development files Expat is an XML parser library written in C. It is a stream-oriented parser in which an application registers handlers for things the parser might find in the XML document (like start tags). . This package provides development files for Expat. Next is `xmlwf`, which should also depend on `libexpat.1` since the `xmlwf` utility is dynamically linked against the `libexpat.so.1` library. Architecture: any Platform: all Depends: libexpat.1 Description: XML parser library - example application This package provides an example application of Expat that determines if an XML document is well-formed. Finally, we can write metadata for `xmlwf-doc`, which should depend on `xmlwf` since it documents the `xmlwf` utility. Architecture: all Platform: all Depends: xmlwf Description: XML parser library - example application documentation files This package provides the manual page for xmlwf, an example application of Expat that determines if an XML document is well-formed. Binary Package Data Files ------------------------- The **oh-installfiles**(1) utility of opkhelper, which we'll be using to install files into *binary package data directories*, requires a `files` file for each binary package that is to provide data files. Recall how we decided to split files between packages. We will now write pathname patterns to do this. Again, let's start with `libexpat.1`. We can write the following pattern in `libexpat.1.pkg/files`: /usr/lib/*/libexpat.so.* This will match `/usr/lib/core-linux-eglibc/libexpat.so.1` and `/usr/lib/core-linux-eglibc/libexpat.so.1.6.0`; these two files will be provided by `libexpat.1`. The patterns for `libexpat.1-dev` are a little more complicated: /usr/include /usr/lib/*/libexpat.so /usr/lib/*/libexpat.a /usr/lib/*/pkgconfig The first pattern simply matches the directory containing header files. The second matches the versionless symbolic link; remember this is used by **ld**(1) to link a just-compiled object against `libexpat.so.1.6.0`. The third matches the static library, and the fourth matches the directory containing the `expat.pc` pkg-config file. `xmlwf.pkg/files` need only contain a pattern to match the directory containing the `xmlwf` utility. /usr/bin `xmlwf-doc.pkg/files` is similarly simple: /usr/share/man/man1 With these pathname patterns done, we can add **oh-installfiles**(1) to our `build` makefile: #!/usr/bin/make -f nop: @: build: oh-autoconfigure oh-autobuild touch $@ install: build oh-autoinstall oh-installfiles Now run **opkbuild**(1) again: $ opkbuild -b -c -T install You can verify that all files were installed where they should be: $ find tmp/*.data -exec ls -Fd '{}' ';' tmp/libexpat.1.data/ tmp/libexpat.1.data/usr/ tmp/libexpat.1.data/usr/lib/ tmp/libexpat.1.data/usr/lib/core-linux-eglibc/ tmp/libexpat.1.data/usr/lib/core-linux-eglibc/libexpat.so.1@ tmp/libexpat.1.data/usr/lib/core-linux-eglibc/libexpat.so.1.6.0* tmp/libexpat.1-dev.data/ tmp/libexpat.1-dev.data/usr/ tmp/libexpat.1-dev.data/usr/lib/ tmp/libexpat.1-dev.data/usr/lib/core-linux-eglibc/ tmp/libexpat.1-dev.data/usr/lib/core-linux-eglibc/pkgconfig/ tmp/libexpat.1-dev.data/usr/lib/core-linux-eglibc/pkgconfig/expat.pc tmp/libexpat.1-dev.data/usr/lib/core-linux-eglibc/libexpat.so@ tmp/libexpat.1-dev.data/usr/lib/core-linux-eglibc/libexpat.a tmp/libexpat.1-dev.data/usr/include/ tmp/libexpat.1-dev.data/usr/include/expat_external.h tmp/libexpat.1-dev.data/usr/include/expat.h tmp/xmlwf.data/ tmp/xmlwf.data/usr/ tmp/xmlwf.data/usr/bin/ tmp/xmlwf.data/usr/bin/xmlwf* tmp/xmlwf-doc.data/ tmp/xmlwf-doc.data/usr/ tmp/xmlwf-doc.data/usr/share/ tmp/xmlwf-doc.data/usr/share/man/ tmp/xmlwf-doc.data/usr/share/man/man1/ tmp/xmlwf-doc.data/usr/share/man/man1/xmlwf.1 Cleaning Up Installed Files --------------------------- There are few things we can do to improve our `build` makefile's `install` target. You may have noticed **oh-installfiles**(1) warn that something hasn't been installed: > oh-installfiles: Warning: Some files have not been installed into packages With **find**(1), we can see that this is the `libexpat.la` file that GNU libtool generated. $ find tmp/dest -type f -exec ls -Fd '{}' ';' | sed 's|^tmp/dest||' /usr/lib/core-linux-eglibc/libexpat.la* We don't need this, and we can simply delete it in the `install` target. Next, note that some file permissions aren't entirely correct. For example, `libexpat.so.1.6.0` is executable, but almost all libraries need not be. So we can call **oh-fixperms**(1) in our `install` target to automatically set correct permissions for us. Finally, note that the executable and linkable objects are not stripped: they contain all of their symbols, including those only needed for debugging. $ file tmp/libexpat.1.data/usr/lib/core-linux-eglibc/libexpat.so.1.6.0 tmp/libexpat.1.data/usr/lib/core-linux-eglibc/libexpat.so.1.6.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0x2d88e36feeb8245bfa2f63f2f0e9a9f8232f6d2c, not stripped $ file tmp/xmlwf.data/usr/bin/xmlwf tmp/xmlwf.data/usr/bin/xmlwf: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.26, BuildID[sha1]=0xdb5f686930b13b8a5e7519efb446a2da14de9856, not stripped We can call **oh-strip**(1) in our `install` target to automatically strip objects for us. So our `build` makefile should now look like this: #!/usr/bin/make -f nop: @: build: oh-autoconfigure oh-autobuild touch $@ install: build oh-autoinstall rm -f 'dest/usr/lib/$(OPK_HOST_ARCH)/libexpat.la' oh-fixperms oh-strip oh-installfiles TODO: Finish. [opkbuild]: http://git.os.libiquity.com/opkbuild/opkbuild.git/ [spf]: http://specs.os.libiquity.com/spf-2.0/ [opkhelper]: http://git.os.libiquity.com/opkhelper/opkhelper.git/ [posix]: http://pubs.opengroup.org/onlinepubs/9699919799/ [posix-makefile]: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/make.html#tag_20_76_13 [spf-format]: http://specs.os.libiquity.com/spf-2.0/overview.html#files-format [expat]: http://expat.sourceforge.net/ [spf-upstream-source]: http://specs.os.libiquity.com/spf-2.0/overview.html#files-src-src-ver-tar-ext [dpm-control]: http://www.debian.org/doc/debian-policy/ch-controlfields.html#s-controlsyntax [spf-fields-src]: http://specs.os.libiquity.com/spf-2.0/fields.html#fields-src [rfc-5322-3.4]: https://tools.ietf.org/html/rfc5322#section-3.4 [spf-changelog]: http://specs.os.libiquity.com/spf-2.0/metadata.html#changelog [spf-build]: http://specs.os.libiquity.com/spf-2.0/buildsys.html#build [no-op]: https://en.wiktionary.org/wiki/no-op [spf-work-area]: http://specs.os.libiquity.com/spf-2.0/buildsys.html#work-area [posix-colon]: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#colon [spf-binpkg.pkg]: http://specs.os.libiquity.com/spf-2.0/overview.html#files-binpkg.pkg [spf-fields-bin]: http://specs.os.libiquity.com/spf-2.0/fields.html#fields-src