Building Eiffel Shared Libraries on Linux

Introduction & rationale

Shared libraries are an essential technique when building modular systems. A  shared library provides a self contained and well defined API/ABI which can be developed independently and, if proper care is taken, easily be integrated in other modules or systems written in completely different languages and development environments. In particular we are interested in building dynamically shared libraries, in contrast to statically shared libraries.

Shared libraries are often called dynamic libraries. Both these names are used with the same meaning below.

Where to start

In order to build a shared library with EiffelStudio you need:

  • An idea of what functionality you want to implement! :-)
  • A .ecf file describing the Eiffel system.
  • At least one Eiffel class implementing the root class of the system and also containing the implementation of the features of the shared library.
  • A .def file listing the features of the root class that are to be exported.
  • A C header file declaring the exported features. Note: This is not generated by EiffelStudio, even though it could, and maybe should, be.

However due to a few problems with EiffelStudio on Linux you will need to implement some workarounds. The problems are as follows:

These problems existed in EiffelStudio 6.2.

API design

Specifying and designing an API means deciding on two main issues; which features/functions to provide and which types to use.

The mapping of complex Eiffel types to C types is a challenge. Not only because the representation of objects or type instances has to be solved, but also because you need to decide on how you want to present the entire complex object structure inside a running Eiffel system to an outside world that has no knowledge of your Eiffel type system. The approach we prefer is to concentrate on the API as a whole and ask ourselves the question from a potential client's perspective; "what features and what data representations do I want from this API?".

Apart from deciding on what features you want to have in your shared library you need to think carefully about what basic types you want to use in the API entry points. There are two approaches. The first is to provide a typed C API with basic C types and, if you want to make things difficult, C structs. The second is to provide a string based data representation API where you represent basic types in the same way as Python or JSON literals.

The typed C API design

First of all, you need to map Eiffel types to equivalent C types. You can look in the file "eif_types.h" included in the EiffelStudio installation to see how the basic Eiffel types map to C types.

The string based data representation design

With this approach only basic Eiffel types (INTEGER, DOUBLE, STRING) are used and then a serialization mechanism is used for the runtime Eiffel objects (that are instances of complex Eiffel types) that you want to pass or recieve from the outside world. A very good choice for serialization representation is  JSON which can be implemented by Eiffel STRING and C char*. It is readable by Javascript, Python and humans and it is easy to implement JSON parsers for other languages. Here is an  Eiffel library for JSON.

For example, consider the Eiffel class:

  class POINT

  feature 
      x, y, z: INTEGER

Instances of this class can be serialized to:

  {'x': 5, 'y': 8, 'z': 2}

For object structures, like instances of this class:

  class LINE

  feature
    p1, p2: POINT 

we can serialize like this:

  {'p1': {'x': 5, 'y': 8, 'z': 2},
   'p1': {'x': 5, 'y': 8, 'z': 2}}

We can add type tags like "'type': 'LINE'" if we wish too. Assuming that we add unique id:s to the Eiffel objects we serialize we can also serialize complex structures like this:

  {'id': 926563,
   'title': 'Object-Oriented Software Construction',
   'isbn': '0-13-629155-4'
   'author_id': 6433}

and make sure we have an ANSI C API feature:

  char* author_by_id (int);

which returns a serialized representation of an author given a unique author id. For example invoking it with "6433" could return:

  {'id': 6433,
   'name': 'Bertrand Meyer'
   'book_ids': [926563, 425820, 272676]}

API Implementation

Principles

Here we present an example of an approach to implementing C style API:s. In this example we wish to provide a API feature for calulating Fibonacci numbers. For this implementation we have decided on the following:

  1. Only basic types are used in the exported API. Ie. INTEGER, BOOLEAN, POINTER etc.
  2. We use the following convention for returning status codes and results; all features return an API_RESULT in the form of a POINTER to a JSON value. An API_RESULT contains a status code and description and optionally a value.
  3. A serialized API_RESULT has the following structure (EBNF syntax, non terminals in bold face):
    API_RESULT = "[" STATUS "]" | "[" STATUS "," VALUE "]"
    STATUS = STATUS_CODE ":" STATUS_DESCRIPTION
    VALUE = JSON value
    STATUS_CODE = JSON string
    STATUS_DESCRIPTION = JSON string
    
  4. All API queries return an API_RESULT with a value and all API commands return an API_RESULT without a value.

Important things to solve:

  • All internal exceptions must be called at the outmost level.
  • We should take proper care of signal handling, ie. Ctrl-C.

One issue remains to be solved for multithreaded execution; The proposed implementation returns a pointer to a string allocated by the Eiffel system, which thus can be garbage collected. However this will not happen until the next invocation of the API, so the client can safely copy any returned data before that. However in a multithreaded environment another thread may invoke a function in the API and thus trigger the garbage collector which could clean up the string previously returned!

One solution to this problem is to only allow INTEGER status codes to be returned by the API and design all functions that need to return data so that they have a parameter that is a POINTER to a memory space allocated by the caller.

The code

First we have a MATH class containing our fibonacci feature:

class MATH

feature -- Basic operations

    fibonacci (i: INTEGER): INTEGER is
            -- i:th Fibonacci number
        require
            valid_i: i >= 0
        do
            if i = 0 then 
                Result := 0
            elseif i = 1 then
                Result := 1
            else
                Result := fibonacci (i-1) + fibonacci (i-2)
            end
        end

Note that the above class is a normal Eiffel class that could be used in any Eiffel system. We now write our API class:

class MATH_API

feature {NONE} -- Implementation (API for MATH class)

    fibonacci (i: INTEGER): API_RESULT is 
            -- API interface for fibonacci
        local
            code, desc: UC_STRING
        do
            if i < 0 then
                create code.make_from_string ("EPRECF")
                create desc.make_from_string ("Precondition failure in api_fibonacci: i >= 0 [i = " + i.out + "]")
                create Result.make (ucs1, ucs2, Void)
            else
                create Result.make_with_value (Void, Void, fibonacci (i))
            end
        end

    math: MATH is
        once
            Create Result
        end

feature -- Implementation (C API)

    math_fibonacci (i: INTEGER): POINTER is 
            -- C API interface for api_fibonacci. This feature is exported
            -- as a shared library symbol and can be called from C
        local
            s: UC_STRING
            a: ANY
        do
            s := fibonacci (i).json_representation
            a := s.string.to_c
            Result := $a
        end

Multithreaded shared libraries

Threads are evil. Be very very careful with them in shared libraries and make sure you have very very good reasons for using them. They break the deterministic sequential execution of your system and they are notoriously difficult to debug. See:  http://badtux.org/home/eric/editorial/threads.php,  http://best-practice-software-engineering.blogspot.com/2007/12/arch-threads-are-evil.html and  http://www.softpanorama.org/People/Ousterhout/Threads/sld001.htm

Problems and known issues

For a long time EiffelStudio has been able to create shared libraries for both Linux systems (.so) and for Windows (.dll & .lib). This has worked fine up to release 5.7, the last release of version 5. However beginning with 6.0 there are serious problems on Linux.

Attached to this page there is a tar.gz file containing source code for reproducing these problems:

  • Source code for a trivial Eiffel shared library system called 'sum'.
  • A .def file for the Eiffel shared library.
  • A .ace file for building the Eiffel shared library with EiffelStudio 5.6.
  • The compiled .so file built with EiffelStudio 5.6 on a 32-bit x86 Ubuntu 8.04 system with gcc 4.2.4
  • A .ecf file for building the Eiffel shared library with EiffelStudio 6.2.
  • The compiled .so file built with EiffelStudio 6.2 on a 32-bit x86 Ubuntu 8.04 system with gcc 4.2.4
  • Source code for a trivial Eiffel application called 'client' that uses the Eiffel shared library.
  • A .ecf file for building the Eiffel application with EiffelStudio 6.2
  • Source code for a trivial C application called 'cclient' that uses the Eiffel shared library.
  • A Makefile for building the C application and for installing the shared library built with 5.6 and 6.2 respectively.

Problem 1: Incorrect initialization of the Eiffel runtime

Symptom

A segmentation violation occurs when any function in a dynamic library is called. This does not occur in EiffelStudio 5.6. It has been confirmed by Seibo to occur in 6.1 and 6.2.

This is the generated C code in 6.2 (EIFGENs/sum/F_code/E1/edynlib.c):

/*****************
 *** EDYNLIB.C ***
 *****************/

#include "egc_dynlib.h"

/***************************
 * SHARED_LIBRARY (make) : sum <F22_192> 
 ***************************/
extern void F22_191 (EIF_REFERENCE);
extern EIF_INTEGER_32 F22_192 (EIF_REFERENCE, EIF_INTEGER_32 , EIF_INTEGER_32 );
EIF_INTEGER_32 sum (EIF_INTEGER_32 i, EIF_INTEGER_32 j)
{       GTCX
        /* Creation : F22_191; */
        /* Feature  : F22_192 ;*/
        EIF_REFERENCE main_obj = (EIF_REFERENCE) 0;
        EIF_INTEGER_32 Return_Value ;
        DYNAMIC_LIB_RT_INITIALIZE(1);

RTLR(0,main_obj);
        main_obj = RTLN(21);
        /* Call the creation routine */
        F22_191(main_obj);

        /* Call the routine */
        Return_Value = (EIF_INTEGER_32) F22_192(main_obj,i,j);
        DYNAMIC_LIB_RT_END;
        return (EIF_INTEGER_32) Return_Value;
}

The segmentation fault occurs in the line: main_obj = RTLN(21);

Diagnosis

The Eiffel runtime system is not initialized correctly in the shared library .so file which causes the system to crash with a segmentation violation.

Background

It stopped working in 5.7 because a general issue with multithreaded DLLs was fixed by Eiffel Software sometime during 2006 or 2007 (threads are evil). The fix resulted in the file "egc_dynlib.h" being modified. This file contains a C macro for initializing the Eiffel runtime and looked like this in 5.6:

#ifndef _egc_dynlib_h_
#define _egc_dynlib_h_

#include "eif_cecil.h"
#include "eif_eiffel.h"
        
#ifdef __cplusplus
extern "C" {
#endif

/* Initialization and destruction of runtime.
 * Both routines are defined in `egc_dynlib.c' */
extern void init_rt(void);
extern void reclaim_rt(void);
        
#define DYNAMIC_LIB_RT_INITIALIZE(x)\
        init_rt(); \





        { \
                RTLD; \
                RTLI(x); \
                
#define DYNAMIC_LIB_RT_END \
                RTLE; \
        }
                        
#define DYNAMIC_LIB_RT_RECLAIM(x) \
        reclaim_rt(); 
                        
#ifdef __cplusplus
}
#endif

#endif

In 6.2 the same file "egc_dynlib.h" looks like this:

#ifndef _egc_dynlib_h_
#define _egc_dynlib_h_

#include "eif_cecil.h"
#include "eif_eiffel.h"
        
#ifdef __cplusplus
extern "C" {
#endif

/* Initialization and destruction of runtime. */
#define DYNAMIC_LIB_RT_INITIALIZE(x)\
        { \
                RTGC; \
                EIF_ENTER_EIFFEL; \
                { \
                        RTLD; \
                        RTLI(x); \
                
#define DYNAMIC_LIB_RT_END \
                        RTLE; \
                } \
                EIF_EXIT_EIFFEL; \
        }

#ifndef EIF_WINDOWS
        /* Define calling convention type so that it Eiffel dlls for windows can also be compiled
         * on other platforms where it does not matter. */
#ifndef __stdcall
#define __stdcall
#endif

#ifndef __cdecl
#define __cdecl
#endif

#ifndef __fastcall
#define __fastcall
#endif

#endif

#ifdef __cplusplus
}
#endif

#endif

Note that the call to init_rt () has been removed! The reason for this, according to Eiffel Software, is that if you want the DLL to work in a multithreaded mode with Eiffel, you need to do some manual work. On Windows, there are some hooks which let you know when the DLL is created and released, and when threads are created and released. Because those hooks are missing on Unix, you have to do them manually in the client code of the shared library.

Solution

For non-multithreaded shared libraries, add the init_rt call to the C macro DYNAMIC_LIB_RT_INITIALIZE(x) in the 'egc_dynlib.h' file in your installation:

/* Initialization and destruction of runtime. */
#define DYNAMIC_LIB_RT_INITIALIZE(x)\
        { \
                init_rt (); \
                RTGC; \
                EIF_ENTER_EIFFEL; \
                { \
                        RTLD; \
                        RTLI(x); \
                
#define DYNAMIC_LIB_RT_END \
                        RTLE; \
                } \
                EIF_EXIT_EIFFEL; \
        }

Note: There is currently no documented way to create multithread shared libraries on Linux.

Problem 2: Symbol clashes due to use of global C symbols

Symptom

Name clashes when calling Eiffel shared libraries from other Eiffel systems. [TBD: Add listing of name chashes as example]

Diagnosis

All symbols are exported from the resulting shared library .so file. EiffelStudio generates ANSI C code and a huge number of C functions with names like 'd4x56lk' etc. All these symbols/functions are exported as globals.

Background

This is done because EiffelStudio generates lots of C files that use symbols defined in lots of other C files and it would be difficult to keep track of exactly which C files contains the symbols needed. Thus all symbols are made global.

This is ugly from the perspective of the user of the .so file and will also result in name clashes if you try to link an Eiffel executable with an Eiffel .so since they will both have symbols/functions with the same name. The .so file should only export the symbols explicitly listed in the .def file.

This problem existed in 5.6 too.

Solution

Explicitly list the features (symbols) to be exported using a ld script .map file a with VERSION command, in which you explicitly set which features (symbols) in the final .so file are to be global and which are to be local. This .map file can then be used by invoking the linker with:

  $ ld --version-script foo.map ...

For more info on the ld VERSION command, see:  http://sourceware.org/binutils/docs-2.18/ld/VERSION.html#VERSION.

Note that this solution works for GNU ld on ELF binary systems. Since around 2000, ELF is used on all Linux/Unix? systems on x86 as well as a large number of other Unix/linux systems on other hardware platforms. This means the solution works on probably all Linux/x86 systems and most Unix/x86 systems as well as a large number of other Unix/Linux? systems on other hardware. For an overview see:  http://en.wikipedia.org/wiki/Executable_and_Linkable_Format. Of course, if you are using another C compiler/linker toolchain than GNU gcc/ld you will need to find another solution.

Manual solution

  1. Go to the root directory of your Eiffel project and create a .map file. Here is an example from the attached dynlib-test library:
VERS_1 {
         global:
                 sum;
         local:
                 *; 
};

In the file above we specify that the feature 'sum' is to be made global, ie. exported, and all other features are to be made local, by use of the wild card symbol '*'. Since we actually only want to control the visibilty of symbols, and not use the ABI (Application Binary Interface) version control features of the ld VERSION command, we don't really need an explict version declaration like 'VERS_1'. However it's a common convention and it doesn't do any harm, so we'll put it in!

Note that the list of features (symbols) that we need to specify in the 'global' section should be the same exact list as the features listed in the EiffelStudio .def file for the shared library!

  1. Copy the .map file file to the F_code directory of your Eiffel project.
  1. Go to the F_code directory of your Eiffel project and edit the Makefile.SH file add the flag --version-script <NAME_OF_SCRIPT_FILE> to LDSHAREDFLAGS. Here is an example from the attached dynlib-test library:
LDSHAREDFLAGS = --version-script sum.map $ldsharedflags

Note that '$ldsharedflags' should already be in the Makefile.SH file!

  1. In the F_code directory run the EiffelStudio command:
$ finish_freezing

Automated solution

This is automatically taken care of if you build your Eiffel shared libraries with the modified version of Peter Gummer's SCons Eiffel tool. The modified version parses the .def file and generates a .map file, and then also modifies the Makefile.SH so that the linker is invoked with the correct flags.

Problem 3: Dead code removal removes the exported features

Symptom

You get unresolved symbol errors like the following when trying to call features in your shared libraries:

Unresolved symbol: T345_675

Diagnosis

When you finalize shared library systems the compiler removes all features in the shared library since they are not in any explicit call chain from the root feature in the root class.

Background

Solution

A hack solution is to add a reference to all your features in the creation feature of the class containing the exported features inside a conditional branch that never is executed. Here is an example from the attached dynlib-test example:

indexing
    description : "Root class for testing to build a shared library on Linux"
    date: "$Date$"
    revision: "$Revision$"

class
    SHARED_LIBRARY

create
    make

feature -- Initialization

    make is
            -- Run application.
        local
            i: INTEGER
        do
            if False then
                i := sum (0, 0)
            end
        end

feature -- Exported features

    sum (i, j: INTEGER): INTEGER is
            -- Return the sum of i and j.
        do
            result := i + j
        end

end -- class SHARED_LIBRARY

Problem 4: Handling CTRL-C

Symptom

When using CTRL-C with an application that uses a Eiffel shared library it will cause an Eiffel stack trace to be dumped.

Diagnosis

Importing the above described 'pysum.py' module, calling 'pysum.sum ()' in the Python Intrepreter and then pressing CTRL-C produces:

sum: PANIC: main entry point vanished ...

sum: system execution failed.
Following is the set of recorded exceptions.
NB: The raised panic may have induced completely inconsistent information:

-------------------------------------------------------------------------------
Class / Object      Routine                Nature of exception           Effect
-------------------------------------------------------------------------------
RUN-TIME            root's creation        Interrupt:                   
<00000000B76DEC94>                         Operating system signal.      Exit
-------------------------------------------------------------------------------
RUN-TIME            root's creation                                     
<00000000B76DEC94>                         Routine failure.              Exit
-------------------------------------------------------------------------------
RUN-TIME            root's creation        Segmentation fault:          
<00000000B76DEC94>                         Operating system signal.      Exit
-------------------------------------------------------------------------------
RUN-TIME            root's creation        main entry point vanished:   
<00000000B76DEC94>                         Eiffel run-time panic.        Bye
-------------------------------------------------------------------------------

sum: dumping core to generate debugging information...

sum: PANIC CASCADE: Unexpected harmful signal (Segmentation fault) -- Giving up...

Background

Solution

Generating Python wrappings with SWIG

For information on SWIG see:  http://www.swig.org/Doc1.3/SWIGDocumentation.html.

It is nice to generate Python wrappings of a shared library so that you can easily try out and test your shared library with the interactive power of a scripting language, in this case Python. In the description and example below we assume that the shared library example from the attached dynlib-test library has been compiled and installed as 'libsum.so'. This can be done with:

$ cp ./EIFGENs/sum/F_code/sum.so /usr/local/lib/libsum.so
$ ldconfig;   
  1. Write a SWIG interface .i file. Here is an example file sum.i from the attached dynlib-test library:

#!c
%module pysum
%{
#include "sum.h"
%}
  1. Run swig from the command line:
$ swig -python sum.i

This will create the following files

pysum.py
sum_wrap.c

Where 'pysum.py' is the Python module and 'sum_wrap.c' contains the generated C wrapper code for the Python module.

  1. Compile and link the generated C wrapper code. This can be done either via distutils or by hand or finally with SCons. Here we do it by hand to show what needs to be done (and what distutils and SCons do behind the scenes). Run the following command:
$ gcc -c -fPIC sum_wrap.c -I/usr/include/python2.5

This will generate the object file 'sum_wrap.o. Finally we need to link it and create a .so file with:

$ gcc -shared sum_wrap.o -lsum -o _pysum.so

Note that we need to generate a .so file with the same name as our Python module but with a leading underscore character '_'. This is the standard Python convention.

  1. You can test this by running:
$ python
Python 2.5.2 (r252:60911, Jul 31 2008, 17:28:52) 
[GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pysum
>>> pysum.sum (7, 8)
>>> 15
>>> <CTRL-D>

Building shared libraries with SCons

For general information on using SCons to build Eiffel systems read this first.

Building shared libraries with the SCons EiffelStudio Tool is as easy as building ordinary programs. This can be easily achieved using SCons' built in support for compiling shared libraries with gcc or MSVS and for SWIG. The SConscript file below also makes sure to copy the finalized shared library from the EIFGENs/sum/f_code library to the current working directory, which in this case is the same directory as the Scons files and the .ecf file is located. Finally the Sconscript file ensures that it works well on both Linux and Windows.

Here is the example SConstruct and SConscript file for achieving the above for the attached dynlib-test library:

SConstruct

import os

# Create a SCons environment with the SCons Eiffel Tool
env = Environment(ENV = os.environ, tools = ['default', 'Eiffel'])
env.Append (arguments = ARGUMENTS)

SConscript ('SConscript', 'env')

Sconscript

import os, sys

Import ('env')

module_name = 'sum'
py_module_name = 'py' + module_name
ecf_file_name = module_name + '.ecf'
swig_file = module_name + '.i'
rc_file = module_name + '.rc'
res_file = module_name + '.res'

if sys.platform == "win32":
    bin_name = module_name + '.dll'
else:
    bin_name = module_name + '.so'
    so_name = 'lib' + bin_name

eif_targets = env.Eiffel (target = bin_name, source = ecf_file_name)
#print "eif_targets: " + str (eif_targets)

libsum_target = env.Command (so_name, str (eif_targets[0]), Copy ("$TARGET", "$SOURCE"))
#print "libsum_target: '" + str (libsum_target) + "'"

env.Append (SWIGFLAGS = '-python')
if sys.platform == "win32":
    env.RES (target = res_file, source = rc_file)
    env.Append (SWIGFLAGS = '-I.')
    env.Append (CPPPATH = ['.'])
else:
    env.Append (CPPPATH = '/usr/include/python2.5')
if sys.platform == "win32":
    env.SharedLibrary (target = py_module_name, source = [swig_file, res_file, bin_name])
else:
    env['LIBS'] = [module_name]
    env['LIBPATH'] = ["."]
    env['SHLIBPREFIX'] = ["_"]
    shlib_targets = env.SharedLibrary (target = py_module_name, source = [swig_file])
    #print "shlib_targets: " + str (shlib_targets)

Attachments