Copyright (c) 2003, Philip Nye, Engineering Arts UK. philipn@engarts.com

Introduction

uClinux uses flat-binary (bFLT) format executables. The bFLT format is used because the kernel cannot juggle memory management to make binaries appear where it wants so the loader has to be able to relocate the code quickly and easily to run where it is loaded.

For further reading on this see the FAQ on Execute in Place (XIP). Also look at the code which loads flat files in linux-2.4.x/fs/binfmt_flat.c and the format definitions in linux-2.4.x/linux/flat.h

In preparing the executable to run, the kernel has to fix up a list of references in the code according to where the executable is to be located. These may be internal links to text, data or BSS sections which can be in different places depending on whether the executable is executed in place or not.

The combination of bFLT format and XIP provides most of what is required for sharing libraries - the system already has the tools to resolve references in the code and to run the code in the same place, but with different data sections.

The final extension for sharing a library is to add a library ID code to the list of references in an executable. This ID is contained in the high byte of the reference. At run time each library is identified by a numeric ID code in the range 1..(MAX_SHARED_LIBS-1) (IDs of 0 or 0xff do not refer to external libraries but are used for other purposes). When the loader encounters a reference with an ID different to that of the current executable, it looks for the library of that ID and if necessary loads it (recursively loading any libraries that one needs) before continuing to resolve the reference.

Naming shared libraries.

Shared library binaries are simple bFLT files the same as any other executable. When it needs library with ID=N the loader looks for the file "/lib/libN.so" and uses the loader to load it. If you want to store a shared library somewhere else or give it a different name, (e.g. /mnt/nfsshare/mylib.so) you can put a symbolic link in the romfs:

	$ ln -s /mnt/nfsshare/mylib.so romfs/lib/libN.so

Creating shared libraries.

When creating and using shared libraries, a few of the flags to the compiler need to change. In particular the -msep-data flag should be replaced with "-mid-shared-library -mshared-library-id=0" when compiling .c files. Other changes can be seen in vendors/config/m68knommu/config.arch (look for BUILD_SHARED).

e.g. to compile a shared library mylib which will have library ID 3 from mylibA.c and mylibB.c - first make a normal library archive:

    m68k-elf-gcc -g -mid-shared-library \
        [other flags definitions and includes] \
        -c -o mylibA.o mylibA.c

    m68k-elf-gcc -g -mid-shared-library \
        [other flags definitions and includes] \
        -c -o mylibB.o mylibB.c

    ar -r libmylib.a mylibA.o mylibB.o

Now link the library executable:

    m68k-elf-gcc -o libmylib -g \
       -nostartfiles  -mid-shared-library \
       [other flags definitions and includes] \
       -Wl,-elf2flt -nostdlib \
       -Wl,-shared-lib-id,3 \
       ${UCLINUXDIST}/uClibc/lib/main.o \
       -Wl,--whole-archive,libmylib.a,-lgcc,--no-whole-archive

Note: the flags "-Wl,-shared-lib-id,3" fixes the library ID as 3. The file ${UCLINUXDIST}/uClibc/lib/main.o is a special shared library entry. This link produces a debug file "libmylib.gdb" which we will need later.

You may need to hide some of the symbols in the library which you don't want shared by making them local. Paul Dale at Snapgear writes: "We take out a pile of symbols that don't make a lot of sense for the library and the main application to share. Things like __main, _start and friends are defined in libc but cannot be shared between library and application because they perform needed initialisation of local state. Hence these symbols must be hidden from the application so the library works yet still be present so the application does. We turn them into local symbols in the library and then link against the non-shared library to get them for the main application.

Some other symbols we hide are tool chain generated GOT and library ID from memory."

    m68k-elf-objcopy \
      -L _GLOBAL_OFFSET_TABLE_ \
      -L main \
      -L __main \
      -L lib_main \
      -L __do_global_dtors \
      -L __do_global_ctors    \
      -L __CTOR_LIST__ \
      -L __DTOR_LIST__        \
      -L _current_shared_library_a5_offset_        \
      libmylib.gdb

    cp libmylib romfs/lib/lib3.so

Using shared libraries

Now you have the shared library (or perhaps it has come from somewhere else) you need to be able to compile your programs to use it. When you link your program, the linker needs to know where all the external symbols are and if necessary turn them into shared library references in the bFLT binary. With ELF shared libraries all the necessary symbol information is there in the library itself, but the bFLT format does not include symbol information. However, all the necessary information IS in the .gdb file which was produced at the same time as the library.

So to compile your program "myprog" which uses the shared library "libmylib" you need to do two things:

1. Use the flags -mid-shared-library -mshared-library-id=0 when compiling:

       m68k-elf-gcc -g -mid-shared-library -mshared-library-id=0 \
           [other flags definitions and includes] \
           -c -o myprog.o myprog.c
   

2. To resolve references when linking you need to tell the linker to look in libmylib.gdb before it looks at the library .a file:

       m68k-elf-gcc -g -o myprog \
           -mid-shared-library -mshared-library-id=0 \
           [other flags definitions and includes] \
           -Wl,-elf2flt -Wl,-shared-lib-id,0 \
           -nostartfiles ${UCLINUXDIST}/uClibc/lib/crt0.o \
           myprog.o \
           -Wl,-R,libmylib.gdb -lmylib
   

The flags "-Wl,-R,libmylib.gdb" do the trick. The flag "-lmylib" make the linker statically link anything which was left unresolved.

Your executable myprog should now run against the shared library. The symbol file libmylib.gdb is not needed at run time and does not go into romfs.

If the library gets re-compiled, you will also need to recompile the executable to take account of the moved symbols. This is different from ELF shared libraries where the calling program only needs recompiling if the interface changes. The loader checks on the build date when loading libraries and refuses to load a library which is younger than the executable which refers to it.

If your library refers to other libraries (e.g. to uClibc) you can apply the same techniques recursively. e.g when linking your library you would need to add "-Wl,-R,libc_path/libc.gdb -lc".

If you want to share uClibc and you are building from the uClinux distribution level you do not need to turn on shared library support (HAVE_SHARED) in uClibc's config file. You just need to turn on CONFIG_BINFMT_SHARED_FLAT in the kernel configuration.

Limit on Numbers of Shared Libraries

The limit on the number of shared libraries is MAX_SHARED_LIBS which is defined in linux-2.4.x/include/linux/flat.h where it is set to 4. However, library ID 0 is not really a shared library which leaves 3 as Pauli says. It would be very easy to change.

Note from Paul Dale at Snapgear: "Make sure you're aware of the consequences that changing this value will have.

Every data segment includes this many entries (at four bytes each) at its beginning regardless of their being used or even needed. Not such a big deal? Perhaps. A data segment is allocated not only for the main program but for each library it uses. In our products almost every process has two data segments (one for the program and one for libc). A few have more. Setting MAX_SHARED_LIBS high will eat your memory nice and quick.

Another catch is that the maximum value for this is 256 due to the way I encoded library addresses. Sure this can be changed but it requires quite a bit more effort :-)"

Resource and Performance Issues

The benefits of shared vs statically linked libraries in an embedded uClinux syatem are a complex issue. Here are some pointers.

If a shared library is used, then the entire library is present in the target whereas if the library is statically linked, then only those routines actually used are included - however they are duplicated once for each program that uses them.

Each shared library an application uses is allocated it's own data segment (per application). In some circumstances (e.g. if dynamically assignable RAM is scarce while code is run XIP from ROM) this could make shared libraries more "costly" than statically linked ones.

The additional linking necessary could make an application take longer to start up using a shared library. This is a small effect but could be noticeable in a system where many apps are run frequently and briefly or where a very rapid response is needed.

Finally, the GOT indirections used for shared libraries are slightly slower than XIP, which is slightly slower than fully relocated executables with no XIP.