Copyright © 2010-2016 magicErmine
Table of Contents
List of Examples
Have you ever asked yourself:
How do I compile a static binary?
Can I force static linking of shared libraries?
How can I statically link an existing Linux executable?
Can I convert a shared libray to a static library?
If the answer is yes, then the question you are actually asking is: How can I make my Linux executable portable?
This book describes creating portable Linux executables using Ermine.
Table of Contents
ErminePro:
$ ErminePro [option]... <orig_exe1> [ <orig_exe_2> ...]
ErmineLight:
$ ErmineLight [option]... <orig_exe>
Below is the list of ErminePro and ErmineLight options. Any option available for ErmineLight is also available for ErminePro. Options that will only work with ErminePro or starting from specific version are marked as such.
ErminePro and ErmineLight Options
- -h, --help
Display a short help message
- --help-options
Display the list of available options
- --help-config [ErminePro]
Display information about the config file format
- --help-tutorial
Dispaly a short explanation on what some of the options do and how they are used
- -c, --config=<config_file> [ErminePro]
Specify config file
- -k, --keep-working-directory
Do not remove working directory ($HOME/.ermine/pack)
- -K, --kernel-version
Show the minimum kernel version to be able to run orig_exe
- -a, --ld_assume_kernel=<kernel>
Set
LD_ASSUME_KERNEL- -l, --ld_library_path=<LD_LIBRARY_PATH>
Set
LD_LIBRARY_PATH- -p, --ld_preload=<LD_PRELOAD>
Set
LD_PRELOAD- -m, --max_ifd=<NUMBER> [ErminePro] [3.1.1]
Set maximum file descriptor (internal file descritor) available to the packaged file to NUMBER
- -o, --output=<new_exe>
Specify output file. This switch is mandatory unless
--kernel-versionis specified.- --with-gconv=TYPE
Specify how to handle the gconv libraries. TYPE is one of
ignore
internal
external
noentry
- --with-locale=TYPE [ErminePro]
Specify how to handle locale files
- --with-xlocale=TYPE [ErminePro]
Specify how to handle locale files used by X
- --with-nss=TYPE
Specify how to handle NSS libraries
Table of Contents
While some options, like --output,
--keep-working-directory or --help
are obvious and self-explanatory
there are others options that may be less obvious.
Let's take a closer look at these options.
Some programs do all the work by themselves - such as dd, ls, rm and many others. But more complicated programs often invoke helper programs; for example, gcc uses:
- cc1
preprocessor + compiler
- as
assembler
- collect2
linker
- ld
real linker, invoked by collect2
So as an example let's pack gcc in a simple way:
$ ErminePro /usr/bin/gcc --output=gcc.ermine
gcc.ermine will be packed successfully and will run on the system where it was packed. But when we're going to move it to the other machine we will encounter a problem: gcc.ermine will be able to run and to parse passed options, but it will fail to run its helper programs -- at best. At worst it will run the ones it finds on the host box, and in all probability the outcome will likely be different from what is desired.
The "simple" way, while having the obvious advantage of being simple, often does not produce the results we are looking for -- thus we have to resort to using multiple input executables.
$ ErminePro /usr/bin/gcc /usr/libexec/gcc/i386-redhat-linux/3.4.2/cc1 /usr/bin/as /usr/libexec/gcc/i386-redhat-linux/3.4.2/collect2 /usr/bin/ld --output=gcc.ermine
When invoked this way, Ermine will pack all listed executables and all shared libraries needed by each one of the executables.
Note
When packing multiple executables with Ermine it's important to specify the "main" executable first. All other executables may be listed in any order.
Specifying a config file allows us to include additional files when packaging, or change the way Ermine-packaged executable will handle specified files.
The Ermine Config file format is very simple:
-
Lines beginning with
#are comments. - Empty lines are ignored.
-
Any other line should be in format:
FileName TYPE
FileName is the name of the file - it can either be absolute or relative (relative to the current working directory at packaging time).
TYPE should have one of the following values:
- external
Ermine-packed executable should access this file on the real file system and not "from within itself".
- internal
This file should be packed, and the Ermine-packed executable should access this file "from within itself".
- noentry
The Ermine-packed executable should pretend that this file doesn't exist. Any attempt to access it (access, open, stat, execve, etc) will return an ENOENT error.
If TYPE is internal
then the file given via FileName should be of one of the following types:
- regular file
- directory
- link
If the file is a link then it should point to a regular file or directory.
If the file doesn't exist it will be packaged as a
noentry file without any message.
If this switch is specified then Ermine will print the minimal kernel version needed to run the created Ermine-packed executable to stdout.
This flag is named after the environment variable
LD_ASSUME_KERNEL used by ld-linix.
Here is an excellent article written by Ulrich Drepper
on LD_ASSUME_KERNEL:
Explaining LD_ASSUME_KERNEL
In short: if there are a number of kernel-dependent libraries installed
(such as libc, libpthread, etc.),
then LD_ASSUME_KERNEL
(and --ld_assume_kernel)
allows us to choose which one to use.
For example, let's try to run ldd on Fedora Core 3 x86 system
with different values for LD_ASSUME_KERNEL:
$ ldd /bin/dd linux-gate.so.1 => (0x00ba6000) libc.so.6 => /lib/tls/libc.so.6 (0x00110000) /lib/ld-linux.so.2 (0x00b40000)
$ LD_ASSUME_KERNEL=2.4.19 ldd /bin/dd linux-gate.so.1 => (0x00980000) libc.so.6 => /lib/i686/libc.so.6 (0x00110000) /lib/ld-linux.so.2 (0x00b40000)
$ LD_ASSUME_KERNEL=2.4.0 ldd /bin/dd linux-gate.so.1 => (0x009a6000) libc.so.6 => /lib/libc.so.6 (0x0081d000) /lib/ld-linux.so.2 (0x00b40000)
Different LD_ASSUME_KERNEL values
will make the program use different libc:
- /lib/tls/libc.so.6
- /lib/i686/libc.so.6
- /lib/libc.so.6
And now run ldconfig:
$ /sbin/ldconfig -p | grep libc.so.6 libc.so.6 (libc6, hwcap: 0x8000000000000000, OS ABI: Linux 2.4.20) => /lib/tls/libc.so.6 libc.so.6 (libc6, hwcap: 0x8000000000000, OS ABI: Linux 2.4.1) => /lib/i686/libc.so.6 libc.so.6 (libc6, OS ABI: Linux 2.2.5) => /lib/libc.so.6
In the example above LD_ASSUME_KERNEL
can change the minimal required kernel from 2.4.20 to 2.2.5
If there are a number of libc libraries installed, the
--ld_assume_kernel flag can be used to allow
running the Ermine-packed application with older kernels.
There is no point using this flag if there is only one libc.
This flag is named after the environment variable
LD_LIBRARY_PATH used by ld-linix.
Like LD_LIBRARY_PATH its value is a colon-separated list
of directories in which to search for ELF libraries
needed by the packed executable
or specified by --ld_preload.
Note
This flag ONLY affects the packaging process, not running of the Ermine-packaged executable.
This flag is named after the environment variable
LD_PRELOAD used by ld-linix.
Like LD_PRELOAD its value is
a space-separated list of shared libraries to be preloaded
and packaged with the application.
Note
This flag ONLY affects the packaging process, not running the Ermine-packaged executable.
But if --ld-preload doesn't affect
running the Ermine-packed executable,
then why bothering specifying it at all?
By default Ermine will only package
the executable itself
and all shared libraries reported by the ldd command.
While this is a good start,
ldd is unable to detect libraries
that are dynamically loadded by an application via dlopen.
So --ld_preload is a way to tell
Ermine that certain libraries need to be packaged too.
Why use the --ld-preload option
instead of just specifying those libraries in the config file?
There are two reasons:
- ErmineLight doesn't offer the
--configoption. - ErminePro has the
--configoption. While it's possible to use a config file instead of the--ld_preloadoption, it is usually inconvenient to do so: often one shared library needs another one, and then in turn they may need more, so tracking all of them can be time-consuming and error-prone.
Example 2.1.
Adding libQtCore.so.4
to a packaged application using --config
libQtCore.so.4 depends on:
$ /usr/lib/libQtCore.so.4 linux-gate.so.1 => (0x006c5000) libpthread.so.0 => /lib/libpthread.so.0 (0x00f5a000) libz.so.1 => /lib/libz.so.1 (0x00b57000) libdl.so.2 => /lib/libdl.so.2 (0x00f39000) libgthread-2.0.so.0 => /lib/libgthread-2.0.so.0 (0x00f0c000) librt.so.1 => /lib/librt.so.1 (0x0068d000) libglib-2.0.so.0 => /lib/libglib-2.0.so.0 (0x00d9c000) libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00110000) libm.so.6 => /lib/libm.so.6 (0x006ec000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x009a3000) libc.so.6 => /lib/libc.so.6 (0x00716000) /lib/ld-linux.so.2 (0x005a6000)Next, a config file should be created:
# config.txt /lib/libpthread.so.0 internal /lib/libz.so.1 internal /lib/libdl.so.2 internal /lib/libgthread-2.0.so.0 internal /lib/librt.so.1 internal /lib/libglib-2.0.so.0 internal /usr/lib/libstdc++.so.6 internal /lib/libm.so.6 internal /lib/libgcc_s.so.1 internal /lib/libc.so.6 internal /lib/ld-linux.so.2 internalAnd finally Ermine should be invoked:
$ ErminePro --config=config.txt ...
Here's another way to achieve the same:
Example 2.2.
Adding libQtCore.so.4
to a packaged application using --ld-preload
$ ErminePro --ld_preload=libQtCore.so.4 ...That's it!
Personally I prefer the latter way, but of course you might find one that suits you better.
This switch allows to specify maximum file descritor available for internal (packaged) file.
By default Ermine reserves space
for 1024 internal file desciptors,
i.e. internal file descriptor can be in the range
[0..1023].
Usually all packaged files opened at application start up,
so even if application uses a lot of file descriptors
default 1024 should be enough.
So when --max-ifd switch should be used?
First, when packaged application aborted with error message like that:
app.ermine: fd_meta_index_set: fd=1057 is not less then limit max_ifd=1024 or negative. Execution aborted.
Obviously, the only way to make this application works is to re-pack it with bigger --max-ifd.
Second, when one wanted to save a bit of memory.
To keep each internal file descriptor 2 long int required.
Default 1024 internal file descriptors will use
8K (2 pages) on i386 system and 16K (4 pages) on x86-64.
1 page on i386 or up to 3 pages on x86-64 may be saved by specifying
smaller value for --max-ifd
Note
Memory for internal file descriptors allocated in whole pages, so --max-fd=5 and --max-fd=256 will result in the same memory usage.This switch allows to control packaging of gconv libraries.
What are gconv libraries?
The standard "libc" library includes the iconv function. This function is used to convert one character set into another. Glibc implemented the iconv function by means of shared objects, each of them providing functions to convert to and from a specific sharacter set.
More information about iconv can be found at: The iconv Implementation in the GNU C library
There are a small number or trivial encodings contained directly in glibc, but most of them are implemented as shared objects. So if an application uses the iconv function then it is likely that it will load gconv libraries as well. Those libraries are tighly coupled with glibc, so mixing different versions of the glibc and gconv libraries is likely to make your application unhappy.
Specifing --with-gconv='internal'
will ensure that the packaged application uses
matching versions of glibc and gconv libraries.
Examples of applications using the iconv function:
- iconv
- bash
- tcsh
This flag allows you to pack localization files. It's a shorcut to adding the following config file line
/usr/lib/locale TYPE
X uses its own locale files to specify locale-specific things, and even worse its own shared libraries to implement them.
The --with-xlocale flag allows you to pack those files.
Of course if the application you are about to pack
is not an X application,
there is no point in using --with-xlocale.
NSS means Name Service Switch. There is a good description of NSS here: http://www.gnu.org/software/libc/manual/html_node/Name-Service-Switch.html
What is important to know about NSS from an application packaging perspective is:
- There are functions such as:
- translation a uid to a user name and back
- translation a gid to a group name and back
- translation a hostname to an IP addresses and back
- and more
- The behavior of those functions is configured in the NSS Configuration File
-
Depending on the configuration
in
/etc/nsswitch.confdifferent NSS shared libraries can be loaded and used by an application, like libnss_files.so.2, libnss_dns.so.2, etc. -
Those libraries aren't detected by ldd,
so if the application uses them
the
--with-nssoption should be specified.
Table of Contents
Below we describe the process of creating a portable executable of PHP on a Fedora Core 3 x86-64 host.
We are going to test the portability on Fedora 12 x86-64.
Maybe we could just copy the php binary from Fedora Core 3 to Fedora 12 and be done with it?
Well, let's try. First just to be sure run the php on Fedora Core 3:
[magicErmine@Fedora3]$ php -v PHP 4.3.9 (cgi) (built: Oct 20 2004 14:07:20) Copyright (c) 1997-2004 The PHP Group Zend Engine v1.3.0, Copyright (c) 1998-2004 Zend Technologies
Looks good.
Now, let's copy the executable to Fedora 12 and try to run it there:
[magicErmine@Fedora12]$ php -v ./php: error while loading shared libraries: libexpat.so.0: cannot open shared object file: No such file or directory
Uh oh! That's not so good. Our simple approach of just copying the executable and hoping it would work has failed -- there is a portability problem.
Let's pack php with the following command:
[magicErmine@Fedora3]$ ErminePro /usr/bin/php -o php.ermine.1
then copy php.ermine.1
to the Fedora12 box and run it there:
[magicErmine@Fedora12]$ ./php.ermine.1 -v PHP Warning: Unknown(): Unable to load dynamic library '/usr/lib64/php4/curl.so' - /usr/lib64/php4/curl.so: cannot open shared object file: No such file or directory in Unknown on line 0 PHP Warning: Unknown(): Unable to load dynamic library '/usr/lib64/php4/fileinfo.so' - /usr/lib64/php4/fileinfo.so: cannot open shared object file: No such file or directory in Unknown on line 0 PHP Warning: Unknown(): Unable to load dynamic library '/usr/lib64/php4/json.so' - /usr/lib64/php4/json.so: cannot open shared object file: No such file or directory in Unknown on line 0 PHP Warning: Unknown(): Unable to load dynamic library '/usr/lib64/php4/ldap.so' - /usr/lib64/php4/ldap.so: cannot open shared object file: No such file or directory in Unknown on line 0 PHP Warning: Unknown(): Unable to load dynamic library '/usr/lib64/php4/mbstring.so' - /usr/lib64/php4/mbstring.so: cannot open shared object file: No such file or directory in Unknown on line 0 PHP Warning: Unknown(): Unable to load dynamic library '/usr/lib64/php4/mcrypt.so' - /usr/lib64/php4/mcrypt.so: cannot open shared object file: No such file or directory in Unknown on line 0 PHP Warning: Unknown(): Unable to load dynamic library '/usr/lib64/php4/mysql.so' - /usr/lib64/php4/mysql.so: cannot open shared object file: No such file or directory in Unknown on line 0 PHP Warning: Unknown(): Unable to load dynamic library '/usr/lib64/php4/mysqli.so' - /usr/lib64/php4/mysqli.so: cannot open shared object file: No such file or directory in Unknown on line 0 PHP Warning: Unknown(): Unable to load dynamic library '/usr/lib64/php4/pdo.so' - /usr/lib64/php4/pdo.so: cannot open shared object file: No such file or directory in Unknown on line 0 PHP Warning: Unknown(): Unable to load dynamic library '/usr/lib64/php4/pdo_mysql.so' - /usr/lib64/php4/pdo_mysql.so: cannot open shared object file: No such file or directory in Unknown on line 0 PHP Warning: Unknown(): Unable to load dynamic library '/usr/lib64/php4/pdo_sqlite.so' - /usr/lib64/php4/pdo_sqlite.so: cannot open shared object file: No such file or directory in Unknown on line 0 PHP Warning: Unknown(): Unable to load dynamic library '/usr/lib64/php4/phar.so' - /usr/lib64/php4/phar.so: cannot open shared object file: No such file or directory in Unknown on line 0 PHP Warning: Unknown(): Unable to load dynamic library '/usr/lib64/php4/sqlite3.so' - /usr/lib64/php4/sqlite3.so: cannot open shared object file: No such file or directory in Unknown on line 0 PHP Warning: Unknown(): Unable to load dynamic library '/usr/lib64/php4/zip.so' - /usr/lib64/php4/zip.so: cannot open shared object file: No such file or directory in Unknown on line 0 PHP 4.3.9 (cgi) (built: Oct 20 2004 14:07:20) Copyright (c) 1997-2004 The PHP Group Zend Engine v1.3.0, Copyright (c) 1998-2004 Zend Technologies
Now at least php is able to run. But where do all those warnings come from?
php.ermine.1 tried (and failed) to load a bunch of dynamic libraries. How does php know what libraries should be loaded?
As can be seen from php's man page
(or from strace -e open /usr/bin/php output)
php uses /etc/php.ini
as config file.
Here is the relevant part of this file from Fedora Core 3:
;;;; ; Note: packaged extension modules are now loaded via the .ini files ; found in the directory /etc/php.d; these are loaded by default. ;;;;
Our first packaging attempt left out two important things:
- the php configuration file
- the php extensions directory
Let's add these to Ermine's configuration file
config.2:
# config.2 /etc/php.ini internal /etc/php.d internal
And then pack them:
[magicErmine@Fedora3]$ ErminePro /usr/bin/php --config=config.2 --output=php.ermine.2
We copy php.ermine.2
to our Fedora 12 machine and run it:
[magicErmine@Fedora12]$ ./php.ermine.2 -v PHP Warning: Unknown(): Unable to load dynamic library '/usr/lib64/php4/ldap.so' - /usr/lib64/php4/ldap.so: cannot open shared object file: No such file or directory in Unknown on line 0 PHP 4.3.9 (cgi) (built: Oct 20 2004 14:07:20) Copyright (c) 1997-2004 The PHP Group Zend Engine v1.3.0, Copyright (c) 1998-2004 Zend Technologies
Now it looks less scary.
If we tak a look at /etc/php.d
on the Fedora Core 3 machine, there is only one entry:
[magicErmine@Fedora3]$ ls /etc/php.d/ ldap.ini
While /etc/php.d
on our Fedora 12 machine has a bit more:
[magicErmine@Fedora12]$ ls /etc/php.d/ curl.ini json.ini mbstring.ini mysqli.ini pdo.ini pdo_sqlite.ini sqlite3.ini fileinfo.ini ldap.ini mcrypt.ini mysql.ini pdo_mysql.ini phar.ini zip.ini
Yes - it's better, but the extensions still can't be loaded.
Lets have a closer look at /etc/php.ini
on Fedora Core 3:
; Directory in which the loadable extensions (modules) reside. extension_dir = /usr/lib64/php4
This directory should be packed too, so we add it to the config file:
# config.3 /etc/php.ini internal /etc/php.d internal /usr/lib64/php4 internal
And now let's package it again:
[magicErmine@Fedora3]$ ErminePro /usr/bin/php --config=config.3 --output=php.ermine.3
Once more we copy php.ermine.3
to our Fedora 12 machine and run it:
[magicErmine@Fedora12]$ ./php.ermine.3 -v PHP Warning: Unknown(): Unable to load dynamic library '/usr/lib64/php4/ldap.so' - libldap-2.2.so.7: cannot open shared object file: No such file or directory in Unknown on line 0 PHP 4.3.9 (cgi) (built: Oct 20 2004 14:07:20) Copyright (c) 1997-2004 The PHP Group Zend Engine v1.3.0, Copyright (c) 1998-2004 Zend Technologies
php.ermine.3, like php.ermine.2
is unable to load ldap.so,
but the reason is different:
php.ermine.2 was unable to find
ldap.so itself,
while php.ermine.3 is unable to find
ldap's dependencies.
While it's possible to chase down the dependencies of all extensions
and add them to the config file it is tedious and error-prone.
A better solution is to just specify all of those libraries with
--ld_preload
switch:
[magicErmine@Fedora3]$ ld_preload=`echo /usr/lib64/php4/*.so` && ErminePro /usr/bin/php --config=config.3 --ld_preload="$ld_preload" --output=php.ermine.4
Now we copy php.ermine.4
to our Fedora 12 machine and run it:
[magicErmine@Fedora12]$ ./php.ermine.4 -v PHP 4.3.9 (cgi) (built: Oct 20 2004 14:07:20) Copyright (c) 1997-2004 The PHP Group Zend Engine v1.3.0, Copyright (c) 1998-2004 Zend Technologies
That's perfect!
Obviously php.ermine.4 will now run
(and that's a good thing),
but php, like a lot of others programs, makes use of
NSS (Network Switch Service)
libraries.
Those libraries weren't packed, so php.ermine.4
is now only running by chance.
Let's add the
--with-nss='internal'
switch.
Finally, here is the command to pack php:
[magicErmine@Fedora3]$ ld_preload=`echo /usr/lib64/php4/*.so` && ErminePro /usr/bin/php --config=config.3 --ld_preload="$ld_preload" --with-nss='internal' --output=php.ermine.5
Now it's really perfect.