path: root/dev/packaging/tutorials/basic.mdwn

[[!meta title="Basic Packaging Tutorial"]]

In this introduction to software packaging, we will package the Expat XML parser
library.  This is a pretty simple but complete package, consisting of a shared
library and its development files plus an executable utility and some
documentation.

We will use [opkbuild 3.0.x][opkbuild] to build a package in [Source Package
Format 2.0][spf] (SPF 2.0) with the assistance of [opkhelper 3.0.x][opkhelper].

This tutorial assumes some knowledge of the UNIX shell command language and
utilities (see the "Shell and Utilities" volume of [POSIX.1-2008][posix]) and at
least basic familiarity with [makefile syntax][posix-makefile].

This tutorial presents one possible packaging workflow that seems to work well.
There is no mandatory workflow to packaging.  The only requirements are those
made by the source package format and any build helper utilities that are used.


Getting Started
===============

Source Package Directory
------------------------

First, make a *source package directory*.  This is the directory that will
contain all of our source package files.  SPF 2.0 makes no requirements on the
name of this directory, but using the name of the source package is recommended.

    $ mkdir expat
    $ cd expat

We need [a file called `format`][spf-format] to identify the format of our
source package.  For SPF 2.0, it should simply contain the string `2.0`.

    $ echo '2.0' >format

Upstream Source Archive
-----------------------

Obviously we need the source code of the software to be packaged.  Go to
[Expat's Web site][expat], find the expat 2.1.0 archive, and download it into
the source package directory.

    $ wget 'http://downloads.sourceforge.net/project/expat/expat/2.1.0/expat-2.1.0.tar.gz'

SPF 2.0 requires that an [upstream source archive][spf-upstream-source] be named
`<pkgname>-<pkgver>.orig.tar<ext>`, where `<pkgname>` is the name of the source
package, `<pkgver>` is the upstream version of the source package, and `<ext>`
is an optional file extension to indicate compression.  So, rename the archive
accordingly.

    $ mv 'expat-2.1.0.tar.gz' 'expat-2.1.0.orig.tar.gz'


Source Package Metadata
=======================

Now we need some metadata for our source package.

Control File
------------

First we'll make a `control` file.  The format of this file is not yet
documented in the SPF 2.0 specification, but it is documented [in the Debian
Policy Manual][dpm-control].  The [source package fields][spf-fields-src] are
`Maintainer` (required), `Build-Depends` (optional), and `Homepage` (optional).
We'll fill in the fields whose values we know right now: `Maintainer` and
`Homepage`.

`Maintainer` is the name and e-mail address of the person or team responsible
for the package (i.e. usually you when you are making a package).  The value
must follow the syntax of the `mailbox` symbol in [RFC 5322 section
3.4][rfc-5322-3.4].  That is, the value must be of the form `name <address>`.
If `name` contains any of the following characters, it must be in double quotes:

    ( ) < > [ ] : ; @ \ , .

`Homepage` is the URL of the Web site for the package, if such a site exists.

Our expat `control` file looks like this:

    Maintainer: "J. Random Hacker" <jrandom@example.com>
    Homepage: http://expat.sourceforge.net/

Change Log
----------

Now we'll make a `changelog` file.  The format of this file is documented [in
the SPF 2.0 specification][spf-changelog].  We're making version "2.1.0-1" of
the "expat" source package for the "trunk" distribution.  We can get the current
date and time in the RFC 5322 format using the **date**(1) command:

    $ LC_ALL='POSIX' date '+%a, %d %b %Y %H:%M:%S %z'

Our expat `changelog` file looks like this:

    expat (2.1.0-1) trunk
    
      * Initial release.
    
     -- "J. Random Hacker" <jrandom@example.com>  Sun, 18 Nov 2012 11:58:19 -0500


Building the Software
=====================

We can now write our `build` makefile to try to get the Expat software to build.
[The `build` makefile][spf-build] "directs the process of building and
installing data files to be provided by binary packages".

Looking Through the Source
--------------------------

With a "[no-op][no-op]" target in `build`, we can make **opkbuild**(1) prepare a
*[build work area][spf-work-area]* with the unpacked source code and stop.  This
target isn't required by SPF 2.0, but it seems to facilitate a nice workflow.
So begin writing `build` as follows:

    #!/usr/bin/make -f
    
    nop:
    	@:

Note that, due to makefile syntax, the line after `nop:` must begin with a tab
character.  This line is called a "command line" in makefile syntax.  The [`:`
utility][posix-colon] is a "null utility" that returns an exit status of zero.
A command prefix of `@` tells make to not write the command to standard output
before executing it.

The `build` makefile must be executable, so set its file mode:

    $ chmod 755 build

We can now make **opkbuild**(1) prepare our build work area.

    $ opkbuild -b -c -T nop

The options are explained in the help output of opkbuild, obtained by running
`opkbuild -h`.  The `-b` option tells **opkbuild**(1) to build only binary
packages (no source package).  The `-c` option tells it to not clean up the work
area after building packages.  The `-T` option specifies a target to be built
instead of the standard `build` and `install` targets.

Now look in `tmp/src/`, the location of the source code within the build work
area.

    $ ls tmp/src/

Look for some documentation file that might tell us how to build Expat.  This
kind of information is usually kept in a file called `INSTALL` or `README`.
Expat's `README` file says to run `./configure`, then `make` and `make install`.

Looking at `tmp/src/configure`, we see that it is "[g]enerated by GNU Autoconf
2.68 for expat 2.1.0".  The `tmp/src/README` file reports that the makefile
supports the use of either the `DESTDIR` or `INSTALL_ROOT` macro to install
Expat somewhere other than in the root of the filesystem.  So, we should be able
to use opkhelper's buildsystem utilities to automatically configure, build, and
install Expat for us.

Building
--------

So let's add a `build` target to our `build` makefile.  The makefile should now
look like this:

    #!/usr/bin/make -f
    
    nop:
    	@:
    
    build:
    	oh-autoconfigure
    	oh-autobuild
    	touch $@

Read the manual pages and/or source code of **oh-autoconfigure**(1) and
**oh-autobuild**(1) to learn more about what they do.

The `touch $@` command is recommended by SPF 2.0:

> The build target should create a file named build in the build work area to
> prevent configuration and compilation from being performed multiple times.

We can now build Expat.

    $ opkbuild -b -c -T build


Installing the Software
=======================

We can now finish our `build` makefile to install the Expat software and make
some binary packages.

Installing
----------

Add a basic `install` target to the `build` makefile.  The makefile should now
look like this:

    #!/usr/bin/make -f
    
    nop:
    	@:
    
    build:
    	oh-autoconfigure
    	oh-autobuild
    	touch $@
    
    install: build
    	oh-autoinstall

The `install` target is declared as depending on the `build` target:

    install: build

Read the manual page and/or source code of **oh-autoinstall**(1) to learn more
about what it does.

Install Expat:

    $ opkbuild -b -c -T install

Splitting Files Into Binary Packages
------------------------------------

Look in the *installation destination directory* `tmp/dest/` for files installed
by Expat's build system.  This can be done with the **find**(1) command, which
results in the following when building for the `core-linux-eglibc` architecture:

    $ find tmp/dest -exec ls -Fd '{}' ';' | sed 's|^tmp/dest||'
    /
    /usr/
    /usr/bin/
    /usr/bin/xmlwf*
    /usr/share/
    /usr/share/man/
    /usr/share/man/man1/
    /usr/share/man/man1/xmlwf.1
    /usr/lib/
    /usr/lib/core-linux-eglibc/
    /usr/lib/core-linux-eglibc/pkgconfig/
    /usr/lib/core-linux-eglibc/pkgconfig/expat.pc
    /usr/lib/core-linux-eglibc/libexpat.so@
    /usr/lib/core-linux-eglibc/libexpat.a
    /usr/lib/core-linux-eglibc/libexpat.la*
    /usr/lib/core-linux-eglibc/libexpat.so.1@
    /usr/lib/core-linux-eglibc/libexpat.so.1.6.0*
    /usr/include/
    /usr/include/expat_external.h
    /usr/include/expat.h

We have the `libexpat.so.1.6.0` shared library and two symbolic links to it:
`libexpat.so.1` and `libexpat.so`.  We have the `libexpat.a` static library and
associated `libexpat.la` library metadata file generated by GNU libtool.  We
have a pkg-config file and two header files.  We have an executable utility and
an associated manual page.

We should therefore split these files into four binary packages: one for the
shared library, one for the library development files, one for the utility, and
one for the utility's documentation.

To find out what we should call the library package, we can use **objdump**(1)
to get the *SONAME* of the library:

    $ objdump -p tmp/dest/usr/lib/core-linux-eglibc/libexpat.so.1.6.0 | grep SONAME
      SONAME               libexpat.so.1

We should name our library package after the SONAME of the shared library,
without `.so`.  The binary package shall be named **`libexpat.1`**.

The versionless `libexpat.so` link is only needed by **ld**(1) when linking a
just-compiled object with the `-lexpat` linker flag.  So this can be provided by
our library development package.  Also provided by that package will be the
header files, the pkg-config file, and the static library.  The development
package can be called **`libexpat.1-dev`**.

The `xmlwf` utility can be provided by a package called simply **`xmlwf`**.

The `xmlwf.1` manual page can be provided by a package called **`xmlwf-doc`**.

Binary Package Metadata
-----------------------

Each binary package to be built needs to have [a directory for its
metadata][spf-binpkg.pkg].  So let's create directories for our packages.

    $ mkdir libexpat.1.pkg libexpat.1-dev.pkg xmlwf.pkg xmlwf-doc.pkg

SPF 2.0 requires a `control` file for each binary package.  The format of this
file is the same as that of the source package `control` file.  The required
[binary package fields][spf-fields-bin] are `Architecture`, `Platform`, and
`Description`.

None of these binary packages are platform-specific, so they will all have a
`Platform: all` field.  All of the binary packages except `xmlwf-doc` are
architecture-specific; that is, they provide files whose contents depend on the
host architecture (files like executable and linkable objects).  So `xmlwf-doc`
will have an `Architecture: all` field while the others will have `Architecture:
any` fields.

Let's start with the `libexpat.1.pkg/control` file:

    Architecture: any
    Platform: all
    Description: XML parser library
     Expat is an XML parser library written in C. It is a stream-oriented parser in
     which an application registers handlers for things the parser might find in the
     XML document (like start tags).

That's fairly simple.

Now let's write a `control` file for `libexpat.1-dev`.  Because it provides
development files for `libexpat.so.1`, `libexpat.1-dev` should depend on the
`libexpat.1` package.  This should be a versioned dependency, because the
`libexpat.so` symbolic link points to a specific version of `libexpat.so`.

    Architecture: any
    Platform: all
    Depends: libexpat.1 (= 2.1.0-1)
    Description: XML parser library - development files
     Expat is an XML parser library written in C. It is a stream-oriented parser in
     which an application registers handlers for things the parser might find in the
     XML document (like start tags).
     .
     This package provides development files for Expat.

Next is `xmlwf`, which should also depend on `libexpat.1` since the `xmlwf`
utility is dynamically linked against the `libexpat.so.1` library.

    Architecture: any 
    Platform: all 
    Depends: libexpat.1
    Description: XML parser library - example application
     This package provides an example application of Expat that determines if an XML 
     document is well-formed.

Finally, we can write metadata for `xmlwf-doc`, which should depend on `xmlwf`
since it documents the `xmlwf` utility.

    Architecture: all 
    Platform: all 
    Depends: xmlwf
    Description: XML parser library - example application documentation files
     This package provides the manual page for xmlwf, an example application of
     Expat that determines if an XML document is well-formed.

Binary Package Data Files
-------------------------

The **oh-installfiles**(1) utility of opkhelper, which we'll be using to install
files into *binary package data directories*, requires a `files` file for each
binary package that is to provide data files.

Recall how we decided to split files between packages.  We will now write
pathname patterns to do this.

Again, let's start with `libexpat.1`.  We can write the following pattern in
`libexpat.1.pkg/files`:

    /usr/lib/*/libexpat.so.*

This will match `/usr/lib/core-linux-eglibc/libexpat.so.1` and
`/usr/lib/core-linux-eglibc/libexpat.so.1.6.0`; these two files will be provided
by `libexpat.1`.

The patterns for `libexpat.1-dev` are a little more complicated:

    /usr/include
    /usr/lib/*/libexpat.so
    /usr/lib/*/libexpat.a
    /usr/lib/*/pkgconfig

The first pattern simply matches the directory containing header files.  The
second matches the versionless symbolic link; remember this is used by **ld**(1)
to link a just-compiled object against `libexpat.so.1.6.0`.  The third matches
the static library, and the fourth matches the directory containing the
`expat.pc` pkg-config file.

`xmlwf.pkg/files` need only contain a pattern to match the directory containing
the `xmlwf` utility.

    /usr/bin

`xmlwf-doc.pkg/files` is similarly simple:

    /usr/share/man/man1

With these pathname patterns done, we can add **oh-installfiles**(1) to our
`build` makefile:

    #!/usr/bin/make -f
    
    nop:
    	@:
    
    build:
    	oh-autoconfigure
    	oh-autobuild
    	touch $@
    
    install: build
    	oh-autoinstall
    	oh-installfiles

Now run **opkbuild**(1) again:

    $ opkbuild -b -c -T install

You can verify that all files were installed where they should be:

    $ find tmp/*.data -exec ls -Fd '{}' ';'
    tmp/libexpat.1.data/
    tmp/libexpat.1.data/usr/
    tmp/libexpat.1.data/usr/lib/
    tmp/libexpat.1.data/usr/lib/core-linux-eglibc/
    tmp/libexpat.1.data/usr/lib/core-linux-eglibc/libexpat.so.1@
    tmp/libexpat.1.data/usr/lib/core-linux-eglibc/libexpat.so.1.6.0*
    tmp/libexpat.1-dev.data/
    tmp/libexpat.1-dev.data/usr/
    tmp/libexpat.1-dev.data/usr/lib/
    tmp/libexpat.1-dev.data/usr/lib/core-linux-eglibc/
    tmp/libexpat.1-dev.data/usr/lib/core-linux-eglibc/pkgconfig/
    tmp/libexpat.1-dev.data/usr/lib/core-linux-eglibc/pkgconfig/expat.pc
    tmp/libexpat.1-dev.data/usr/lib/core-linux-eglibc/libexpat.so@
    tmp/libexpat.1-dev.data/usr/lib/core-linux-eglibc/libexpat.a
    tmp/libexpat.1-dev.data/usr/include/
    tmp/libexpat.1-dev.data/usr/include/expat_external.h
    tmp/libexpat.1-dev.data/usr/include/expat.h
    tmp/xmlwf.data/
    tmp/xmlwf.data/usr/
    tmp/xmlwf.data/usr/bin/
    tmp/xmlwf.data/usr/bin/xmlwf*
    tmp/xmlwf-doc.data/
    tmp/xmlwf-doc.data/usr/
    tmp/xmlwf-doc.data/usr/share/
    tmp/xmlwf-doc.data/usr/share/man/
    tmp/xmlwf-doc.data/usr/share/man/man1/
    tmp/xmlwf-doc.data/usr/share/man/man1/xmlwf.1

Cleaning Up Installed Files
---------------------------

There are few things we can do to improve our `build` makefile's `install`
target.

You may have noticed **oh-installfiles**(1) warn that something hasn't been
installed:

> oh-installfiles: Warning: Some files have not been installed into packages

With **find**(1), we can see that this is the `libexpat.la` file that GNU
libtool generated.

    $ find tmp/dest -type f -exec ls -Fd '{}' ';' | sed 's|^tmp/dest||'
    /usr/lib/core-linux-eglibc/libexpat.la*

We don't need this, and we can simply delete it in the `install` target.

Next, note that some file permissions aren't entirely correct.  For example,
`libexpat.so.1.6.0` is executable, but almost all libraries need not be.

So we can call **oh-fixperms**(1) in our `install` target to automatically set
correct permissions for us.

Finally, note that the executable and linkable objects are not stripped: they
contain all of their symbols, including those only needed for debugging.

    $ file tmp/libexpat.1.data/usr/lib/core-linux-eglibc/libexpat.so.1.6.0
    tmp/libexpat.1.data/usr/lib/core-linux-eglibc/libexpat.so.1.6.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0x2d88e36feeb8245bfa2f63f2f0e9a9f8232f6d2c, not stripped
    $ file tmp/xmlwf.data/usr/bin/xmlwf
    tmp/xmlwf.data/usr/bin/xmlwf: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.26, BuildID[sha1]=0xdb5f686930b13b8a5e7519efb446a2da14de9856, not stripped

We can call **oh-strip**(1) in our `install` target to automatically strip
objects for us.

So our `build` makefile should now look like this:

    #!/usr/bin/make -f
    
    nop:
    	@:
    
    build:
    	oh-autoconfigure
    	oh-autobuild
    	touch $@
    
    install: build
    	oh-autoinstall
    	rm -f 'dest/usr/lib/$(OPK_HOST_ARCH)/libexpat.la'
    	oh-fixperms
    	oh-strip
    	oh-installfiles


TODO: Finish.


[opkbuild]: http://git.os.libiquity.com/opkbuild/opkbuild.git/
[spf]: http://specs.os.libiquity.com/spf-2.0/
[opkhelper]: http://git.os.libiquity.com/opkhelper/opkhelper.git/
[posix]: http://pubs.opengroup.org/onlinepubs/9699919799/
[posix-makefile]: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/make.html#tag_20_76_13
[spf-format]: http://specs.os.libiquity.com/spf-2.0/overview.html#files-format
[expat]: http://expat.sourceforge.net/
[spf-upstream-source]: http://specs.os.libiquity.com/spf-2.0/overview.html#files-src-src-ver-tar-ext
[dpm-control]: http://www.debian.org/doc/debian-policy/ch-controlfields.html#s-controlsyntax
[spf-fields-src]: http://specs.os.libiquity.com/spf-2.0/fields.html#fields-src
[rfc-5322-3.4]: https://tools.ietf.org/html/rfc5322#section-3.4
[spf-changelog]: http://specs.os.libiquity.com/spf-2.0/metadata.html#changelog
[spf-build]: http://specs.os.libiquity.com/spf-2.0/buildsys.html#build
[no-op]: https://en.wiktionary.org/wiki/no-op
[spf-work-area]: http://specs.os.libiquity.com/spf-2.0/buildsys.html#work-area
[posix-colon]: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#colon
[spf-binpkg.pkg]: http://specs.os.libiquity.com/spf-2.0/overview.html#files-binpkg.pkg
[spf-fields-bin]: http://specs.os.libiquity.com/spf-2.0/fields.html#fields-src