Ermine Packagers Guide

Ermine Packagers Guide Multiple HTML Pages

Ermine Packagers Guide Single HTML Page


Introduction to Ermine packaging

Have you ever asked yourself:

  • How do I compile a static binary?

  • Can I force static linking of shared libraries?

  • How can I statically link an existing Linux executable?

  • Can I convert a shared libray to a static library?

If the answer is yes, then the question you are actually asking is: How can I make my Linux executable portable?

This book describes creating portable Linux executables using Ermine.

Chapter 1. Ermine Invocation and Options

Table of Contents

Invocation
Options

Invocation

ErminePro:

$ ErminePro [option]... <orig_exe1> [ <orig_exe_2> ...]

ErmineLight:

$ ErmineLight [option]... <orig_exe>

Note

ErminePro accepts as input multiple executables, but ErmineLight accepts only one.

Options

Below is the list of ErminePro and ErmineLight options. Any option available for ErmineLight is also available for ErminePro. Options that will only work with ErminePro or starting from specific version are marked as such.

ErminePro and ErmineLight Options

-h, --help

Display a short help message

--help-options

Display the list of available options

--help-config [ErminePro]

Display information about the config file format

--help-tutorial

Dispaly a short explanation on what some of the options do and how they are used

-c, --config=<config_file> [ErminePro]

Specify config file

-k, --keep-work-dir

Do not remove working directory ($HOME/.ermine/pack)

-K, --kernel-version

Show the minimum kernel version to be able to run orig_exe

-a, --ld_assume_kernel=<kernel>

Set LD_ASSUME_KERNEL

-l, --ld_library_path=<LD_LIBRARY_PATH>

Set LD_LIBRARY_PATH

-p, --ld_preload=<LD_PRELOAD>

Set LD_PRELOAD

-m, --max_ifd=<NUMBER> [ErminePro] [3.1.1]

Set maximum file descriptor (internal file descritor) available to the packaged file to NUMBER

-o, --output=<new_exe>

Specify output file. This switch is mandatory unless --kernel-version is specified.

--with-gconv=TYPE

Specify how to handle the gconv libraries. TYPE is one of

  • ignore

  • internal

  • external

  • noentry

--with-locale=TYPE [ErminePro]

Specify how to handle locale files

--with-xlocale=TYPE [ErminePro]

Specify how to handle locale files used by X

--with-nss=TYPE

Specify how to handle NSS libraries

Chapter 2. Options in depth

While some options, like --output, --keep-working-directory or --help are obvious and self-explanatory there are others options that may be less obvious. Let's take a closer look at these options.

Multiple input executables [ErminePro]

Some programs do all the work by themselves - such as dd, ls, rm and many others. But more complicated programs often invoke helper programs; for example, gcc uses:

cc1

preprocessor + compiler

as

assembler

collect2

linker

ld

real linker, invoked by collect2

So as an example let's pack gcc in a simple way:

$ ErminePro /usr/bin/gcc --output=gcc.ermine

gcc.ermine will be packed successfully and will run on the system where it was packed. But when we're going to move it to the other machine we will encounter a problem: gcc.ermine will be able to run and to parse passed options, but it will fail to run its helper programs -- at best. At worst it will run the ones it finds on the host box, and in all probability the outcome will likely be different from what is desired.

The "simple" way, while having the obvious advantage of being simple, often does not produce the results we are looking for -- thus we have to resort to using multiple input executables.

$ ErminePro /usr/bin/gcc /usr/libexec/gcc/i386-redhat-linux/3.4.2/cc1 /usr/bin/as /usr/libexec/gcc/i386-redhat-linux/3.4.2/collect2 /usr/bin/ld --output=gcc.ermine

When invoked this way, Ermine will pack all listed executables and all shared libraries needed by each one of the executables.

Note

When packing multiple executables with Ermine it's important to specify the "main" executable first. All other executables may be listed in any order.

--config [ErminePro]

Specifying a config file allows us to include additional files when packaging, or change the way Ermine-packaged executable will handle specified files.

The Ermine Config file format is very simple:

  • Lines beginning with # are comments.
  • Empty lines are ignored.
  • Any other line should be in format:
    FileName TYPE
    

FileName is the name of the file - it can either be absolute or relative (relative to the current working directory at packaging time).

TYPE should have one of the following values:

external

Ermine-packed executable should access this file on the real file system and not "from within itself".

internal

This file should be packed, and the Ermine-packed executable should access this file "from within itself".

noentry

The Ermine-packed executable should pretend that this file doesn't exist. Any attempt to access it (access, open, stat, execve, etc) will return an ENOENT error.

If TYPE is internal then the file given via FileName should be of one of the following types:

  • regular file
  • directory
  • link

If the file is a link then it should point to a regular file or directory.

If the file doesn't exist it will be packaged as a noentry file without any message.

--kernel-version

If this switch is specified then Ermine will print the minimal kernel version needed to run the created Ermine-packed executable to stdout.

--ld_assume_kernel

This flag is named after the environment variable LD_ASSUME_KERNEL used by ld-linix.

Here is an excellent article written by Ulrich Drepper on LD_ASSUME_KERNEL: Explaining LD_ASSUME_KERNEL

In short: if there are a number of kernel-dependent libraries installed (such as libc, libpthread, etc.), then LD_ASSUME_KERNEL (and --ld_assume_kernel) allows us to choose which one to use. For example, let's try to run ldd on Fedora Core 3 x86 system with different values for LD_ASSUME_KERNEL:

$ ldd /bin/dd
	linux-gate.so.1 =>  (0x00ba6000)
	libc.so.6 => /lib/tls/libc.so.6 (0x00110000)
	/lib/ld-linux.so.2 (0x00b40000)

$ LD_ASSUME_KERNEL=2.4.19 ldd /bin/dd
	linux-gate.so.1 =>  (0x00980000)
	libc.so.6 => /lib/i686/libc.so.6 (0x00110000)
	/lib/ld-linux.so.2 (0x00b40000)

$ LD_ASSUME_KERNEL=2.4.0 ldd /bin/dd
	linux-gate.so.1 =>  (0x009a6000)
	libc.so.6 => /lib/libc.so.6 (0x0081d000)
	/lib/ld-linux.so.2 (0x00b40000)

Different LD_ASSUME_KERNEL values will make the program use different libc:

  • /lib/tls/libc.so.6
  • /lib/i686/libc.so.6
  • /lib/libc.so.6

And now run ldconfig:

$ /sbin/ldconfig -p | grep libc.so.6
	libc.so.6 (libc6, hwcap: 0x8000000000000000, OS ABI: Linux 2.4.20) => /lib/tls/libc.so.6
	libc.so.6 (libc6, hwcap: 0x8000000000000, OS ABI: Linux 2.4.1) => /lib/i686/libc.so.6
	libc.so.6 (libc6, OS ABI: Linux 2.2.5) => /lib/libc.so.6

In the example above LD_ASSUME_KERNEL can change the minimal required kernel from 2.4.20 to 2.2.5

If there are a number of libc libraries installed, the --ld_assume_kernel flag can be used to allow running the Ermine-packed application with older kernels. There is no point using this flag if there is only one libc.

--ld_library_path

This flag is named after the environment variable LD_LIBRARY_PATH used by ld-linix. Like LD_LIBRARY_PATH its value is a colon-separated list of directories in which to search for ELF libraries needed by the packed executable or specified by --ld_preload.

Note

This flag ONLY affects the packaging process, not running of the Ermine-packaged executable.

--ld_preload

This flag is named after the environment variable LD_PRELOAD used by ld-linix. Like LD_PRELOAD its value is a space-separated list of shared libraries to be preloaded and packaged with the application.

Note

This flag ONLY affects the packaging process, not running the Ermine-packaged executable.

But if --ld-preload doesn't affect running the Ermine-packed executable, then why bothering specifying it at all?

By default Ermine will only package the executable itself and all shared libraries reported by the ldd command. While this is a good start, ldd is unable to detect libraries that are dynamically loadded by an application via dlopen. So --ld_preload is a way to tell Ermine that certain libraries need to be packaged too.

Why use the --ld-preload option instead of just specifying those libraries in the config file? There are two reasons:

  • ErmineLight doesn't offer the --config option.
  • ErminePro has the --config option. While it's possible to use a config file instead of the --ld_preload option, it is usually inconvenient to do so: often one shared library needs another one, and then in turn they may need more, so tracking all of them can be time-consuming and error-prone.

Example 2.1.  Adding libQtCore.so.4 to a packaged application using --config

First let's see what shared libraries libQtCore.so.4 depends on:
$ /usr/lib/libQtCore.so.4
	linux-gate.so.1 =>  (0x006c5000)
	libpthread.so.0 => /lib/libpthread.so.0 (0x00f5a000)
	libz.so.1 => /lib/libz.so.1 (0x00b57000)
	libdl.so.2 => /lib/libdl.so.2 (0x00f39000)
	libgthread-2.0.so.0 => /lib/libgthread-2.0.so.0 (0x00f0c000)
	librt.so.1 => /lib/librt.so.1 (0x0068d000)
	libglib-2.0.so.0 => /lib/libglib-2.0.so.0 (0x00d9c000)
	libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00110000)
	libm.so.6 => /lib/libm.so.6 (0x006ec000)
	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x009a3000)
	libc.so.6 => /lib/libc.so.6 (0x00716000)
	/lib/ld-linux.so.2 (0x005a6000)
Next, a config file should be created:
# config.txt
/lib/libpthread.so.0     internal
/lib/libz.so.1           internal
/lib/libdl.so.2          internal
/lib/libgthread-2.0.so.0 internal
/lib/librt.so.1          internal
/lib/libglib-2.0.so.0    internal
/usr/lib/libstdc++.so.6  internal
/lib/libm.so.6           internal
/lib/libgcc_s.so.1       internal
/lib/libc.so.6           internal
/lib/ld-linux.so.2       internal
And finally Ermine should be invoked:
$ ErminePro --config=config.txt ...

Here's another way to achieve the same:

Example 2.2.  Adding libQtCore.so.4 to a packaged application using --ld-preload

$ ErminePro --ld_preload=libQtCore.so.4 ...
That's it!

Personally I prefer the latter way, but of course you might find one that suits you better.

--max-ifd [ErminePro] [3.1.1]

This switch allows to specify maximum file descritor available for internal (packaged) file.

By default Ermine reserves space for 1024 internal file desciptors, i.e. internal file descriptor can be in the range [0..1023].

Usually all packaged files opened at application start up, so even if application uses a lot of file descriptors default 1024 should be enough.

So when --max-ifd switch should be used?

First, when packaged application aborted with error message like that:

app.ermine: fd_meta_index_set: fd=1057 is not less then limit max_ifd=1024 or negative. Execution aborted.

Obviously, the only way to make this application works is to re-pack it with bigger --max-ifd.

Second, when one wanted to save a bit of memory. To keep each internal file descriptor 2 long int required. Default 1024 internal file descriptors will use 8K (2 pages) on i386 system and 16K (4 pages) on x86-64. 1 page on i386 or up to 3 pages on x86-64 may be saved by specifying smaller value for --max-ifd

Note

Memory for internal file descriptors allocated in whole pages, so --max-fd=5 and --max-fd=256 will result in the same memory usage.

--with-gconv

This switch allows to control packaging of gconv libraries.

What are gconv libraries?

The standard "libc" library includes the iconv function. This function is used to convert one character set into another. Glibc implemented the iconv function by means of shared objects, each of them providing functions to convert to and from a specific sharacter set.

More information about iconv can be found at: The iconv Implementation in the GNU C library

There are a small number or trivial encodings contained directly in glibc, but most of them are implemented as shared objects. So if an application uses the iconv function then it is likely that it will load gconv libraries as well. Those libraries are tighly coupled with glibc, so mixing different versions of the glibc and gconv libraries is likely to make your application unhappy.

Specifing --with-gconv='internal' will ensure that the packaged application uses matching versions of glibc and gconv libraries.

Examples of applications using the iconv function:

  • iconv
  • bash
  • tcsh

--with-locale [ErminePro]

This flag allows you to pack localization files. It's a shorcut to adding the following config file line

/usr/lib/locale TYPE

--with-xlocale [ErminePro]

X uses its own locale files to specify locale-specific things, and even worse its own shared libraries to implement them.

The --with-xlocale flag allows you to pack those files. Of course if the application you are about to pack is not an X application, there is no point in using --with-xlocale.

--with-nss

NSS means Name Service Switch. There is a good description of NSS here: http://www.gnu.org/software/libc/manual/html_node/Name-Service-Switch.html

What is important to know about NSS from an application packaging perspective is:

  • There are functions such as:
    • translation a uid to a user name and back
    • translation a gid to a group name and back
    • translation a hostname to an IP addresses and back
    • and more
  • The behavior of those functions is configured in the NSS Configuration File
  • Depending on the configuration in /etc/nsswitch.conf different NSS shared libraries can be loaded and used by an application, like libnss_files.so.2, libnss_dns.so.2, etc.
  • Those libraries aren't detected by ldd, so if the application uses them the --with-nss option should be specified.

Chapter 3. Packaging

PHP Packaging

General information

Below we describe the process of creating a portable executable of PHP on a Fedora Core 3 x86-64 host.

We are going to test the portability on Fedora 12 x86-64.

Is PHP packaging really needed?

Maybe we could just copy the php binary from Fedora Core 3 to Fedora 12 and be done with it?

Well, let's try. First just to be sure run the php on Fedora Core 3:

[magicErmine@Fedora3]$ php -v
PHP 4.3.9 (cgi) (built: Oct 20 2004 14:07:20)
Copyright (c) 1997-2004 The PHP Group
Zend Engine v1.3.0, Copyright (c) 1998-2004 Zend Technologies

Looks good.

Now, let's copy the executable to Fedora 12 and try to run it there:

[magicErmine@Fedora12]$ php -v
./php: error while loading shared libraries: libexpat.so.0: cannot open shared object file: No such file or directory

Uh oh! That's not so good. Our simple approach of just copying the executable and hoping it would work has failed -- there is a portability problem.

Packaging -- first attempt

Let's pack php with the following command:

[magicErmine@Fedora3]$ ErminePro /usr/bin/php -o php.ermine.1

then copy php.ermine.1 to the Fedora12 box and run it there:

[magicErmine@Fedora12]$ ./php.ermine.1 -v
PHP Warning:  Unknown(): Unable to load dynamic library '/usr/lib64/php4/curl.so' - /usr/lib64/php4/curl.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning:  Unknown(): Unable to load dynamic library '/usr/lib64/php4/fileinfo.so' - /usr/lib64/php4/fileinfo.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning:  Unknown(): Unable to load dynamic library '/usr/lib64/php4/json.so' - /usr/lib64/php4/json.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning:  Unknown(): Unable to load dynamic library '/usr/lib64/php4/ldap.so' - /usr/lib64/php4/ldap.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning:  Unknown(): Unable to load dynamic library '/usr/lib64/php4/mbstring.so' - /usr/lib64/php4/mbstring.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning:  Unknown(): Unable to load dynamic library '/usr/lib64/php4/mcrypt.so' - /usr/lib64/php4/mcrypt.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning:  Unknown(): Unable to load dynamic library '/usr/lib64/php4/mysql.so' - /usr/lib64/php4/mysql.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning:  Unknown(): Unable to load dynamic library '/usr/lib64/php4/mysqli.so' - /usr/lib64/php4/mysqli.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning:  Unknown(): Unable to load dynamic library '/usr/lib64/php4/pdo.so' - /usr/lib64/php4/pdo.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning:  Unknown(): Unable to load dynamic library '/usr/lib64/php4/pdo_mysql.so' - /usr/lib64/php4/pdo_mysql.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning:  Unknown(): Unable to load dynamic library '/usr/lib64/php4/pdo_sqlite.so' - /usr/lib64/php4/pdo_sqlite.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning:  Unknown(): Unable to load dynamic library '/usr/lib64/php4/phar.so' - /usr/lib64/php4/phar.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning:  Unknown(): Unable to load dynamic library '/usr/lib64/php4/sqlite3.so' - /usr/lib64/php4/sqlite3.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP Warning:  Unknown(): Unable to load dynamic library '/usr/lib64/php4/zip.so' - /usr/lib64/php4/zip.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP 4.3.9 (cgi) (built: Oct 20 2004 14:07:20)
Copyright (c) 1997-2004 The PHP Group
Zend Engine v1.3.0, Copyright (c) 1998-2004 Zend Technologies

Now at least php is able to run. But where do all those warnings come from?

Packaging with php config

php.ermine.1 tried (and failed) to load a bunch of dynamic libraries. How does php know what libraries should be loaded?

As can be seen from php's man page (or from strace -e open /usr/bin/php output) php uses /etc/php.ini as config file.

Here is the relevant part of this file from Fedora Core 3:

;;;;
; Note: packaged extension modules are now loaded via the .ini files
; found in the directory /etc/php.d; these are loaded by default.
;;;;

Our first packaging attempt left out two important things:

  • the php configuration file
  • the php extensions directory

Let's add these to Ermine's configuration file config.2:

# config.2
/etc/php.ini internal
/etc/php.d   internal

And then pack them:

[magicErmine@Fedora3]$ ErminePro /usr/bin/php --config=config.2 --output=php.ermine.2

We copy php.ermine.2 to our Fedora 12 machine and run it:

[magicErmine@Fedora12]$ ./php.ermine.2 -v
PHP Warning:  Unknown(): Unable to load dynamic library '/usr/lib64/php4/ldap.so' - /usr/lib64/php4/ldap.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP 4.3.9 (cgi) (built: Oct 20 2004 14:07:20)
Copyright (c) 1997-2004 The PHP Group
Zend Engine v1.3.0, Copyright (c) 1998-2004 Zend Technologies

Now it looks less scary. If we tak a look at /etc/php.d on the Fedora Core 3 machine, there is only one entry:

[magicErmine@Fedora3]$ ls /etc/php.d/
ldap.ini

While /etc/php.d on our Fedora 12 machine has a bit more:

[magicErmine@Fedora12]$ ls /etc/php.d/
curl.ini      json.ini  mbstring.ini  mysqli.ini  pdo.ini        pdo_sqlite.ini  sqlite3.ini
fileinfo.ini  ldap.ini  mcrypt.ini    mysql.ini   pdo_mysql.ini  phar.ini        zip.ini

Yes - it's better, but the extensions still can't be loaded.

Packaging with extension directory

Lets have a closer look at /etc/php.ini on Fedora Core 3:

; Directory in which the loadable extensions (modules) reside.
extension_dir = /usr/lib64/php4

This directory should be packed too, so we add it to the config file:

# config.3
/etc/php.ini    internal
/etc/php.d      internal
/usr/lib64/php4 internal

And now let's package it again:

[magicErmine@Fedora3]$ ErminePro /usr/bin/php --config=config.3 --output=php.ermine.3

Once more we copy php.ermine.3 to our Fedora 12 machine and run it:

[magicErmine@Fedora12]$ ./php.ermine.3 -v
PHP Warning:  Unknown(): Unable to load dynamic library '/usr/lib64/php4/ldap.so' - libldap-2.2.so.7: cannot open shared object file: No such file or directory in Unknown on line 0
PHP 4.3.9 (cgi) (built: Oct 20 2004 14:07:20)
Copyright (c) 1997-2004 The PHP Group
Zend Engine v1.3.0, Copyright (c) 1998-2004 Zend Technologies

php.ermine.3, like php.ermine.2 is unable to load ldap.so, but the reason is different: php.ermine.2 was unable to find ldap.so itself, while php.ermine.3 is unable to find ldap's dependencies.

Packaging with extension directory and dependencies

While it's possible to chase down the dependencies of all extensions and add them to the config file it is tedious and error-prone. A better solution is to just specify all of those libraries with --ld_preload switch:

[magicErmine@Fedora3]$ ld_preload=`echo /usr/lib64/php4/*.so` && ErminePro /usr/bin/php --config=config.3 --ld_preload="$ld_preload" --output=php.ermine.4

Now we copy php.ermine.4 to our Fedora 12 machine and run it:

[magicErmine@Fedora12]$ ./php.ermine.4 -v
PHP 4.3.9 (cgi) (built: Oct 20 2004 14:07:20)
Copyright (c) 1997-2004 The PHP Group
Zend Engine v1.3.0, Copyright (c) 1998-2004 Zend Technologies

That's perfect!

Perfect?

Obviously php.ermine.4 will now run (and that's a good thing), but php, like a lot of others programs, makes use of NSS (Network Switch Service) libraries. Those libraries weren't packed, so php.ermine.4 is now only running by chance. Let's add the --with-nss='internal' switch.

Finally, here is the command to pack php:

[magicErmine@Fedora3]$ ld_preload=`echo /usr/lib64/php4/*.so` && ErminePro /usr/bin/php --config=config.3 --ld_preload="$ld_preload" --with-nss='internal' --output=php.ermine.5

Now it's really perfect.