The System Assembler

SystemAssembler.py is a Python module that is used to assemble systems. It can be run as a command line program or integrated into SCons as a Scons tool.

SystemAssembler.py reads a formal specification called a System Assembly Specification. The formal specification consists of a single file which by convention uses the file name suffix .sas. Those files are called "sas-files". The syntax of the "sas-file" is a Python literal dictionary.

Every sas-file specifies a system and a list of all its components. Every component is in turn specified by a list of all its component files. For each file it is specified from where and to where it is to be copied. When the method 'assemble' in SystemAssembler is invoked a target directory must be given. This target directory specifies the root directory to where all component files are copied.

The SystemAssembler.py file can be found here.

Rationale

To understand the rationale for the System Assembler one must first understand the Seibo process of building a system and packaging it for installation. The Seibo build process can be outlined as follows:

  1. Build individual binaries. This usually involves compiling compiled software modules to executables or shared libraries, but can also involve other sorts of transformation like producing PDF or Postscript files from Tex or Latex source files etc.
  2. Run module tests. This means running test programs for individual binaries or software libraries.
  3. Assemble all files in the system. This means gathering all files that are to be included in the different system components and laying them out in a directory structure suitable for packaging and/or installation.
  4. Build installation and/or distribution packages. This means packaging the system to a .tar.gz or .zip file or creating .deb or .rpm packages or creating .msi installers.
  5. Run installation tests and system tests.

The System Assembler tool automates step 3 above. The benefits of dividing the build process into these distinct steps are:

  • A clear distinction between the different "build" activities makes the build process easier to understand.
  • It is easier to debug the build system.
  • The build process can easier be split up on different machines.
  • People working with developing programs and modules in the system can work more independently of the people working with developing installation and distribution packages.

In particular the SystemAssembler has the following benefits:

  • The files that actually make up the system are explicitly listed and we thus have a formal specification of the contents of the system that everyone can read and understand.
  • It separates the source code tree from the structure needed by installation packaging tools. This means that the source code tree can be reorganized without having to rewrite the installation packaging scripts.
  • It allows partitioning the system into distinctly packaged components. Different components may still share common files from the system source code.
  • The System Assembler gives explicit warning if files needed by the system are missing.

In many other build processes step 1, 3 and 4 are often not clearly separated which leads to more interdependencies between different build activities. This in turn makes the build system more difficult to develop, maintain and modify. This typically leads to fewer developers in the project have a proper understanding of the "big picture" of the build system.

The sas-file Syntax

The sas-file file syntax is completely based on the Python literal dictionary and is evaluated by the System Assembler as Python code. Therefore you can use normal Python comments in sas-files.

In the formal syntax description below the all non terminals are in capital letters. The special non terminal LITERAL_STRING represents a Python literal string which may be decorated like this 'LITERAL_STRING:[VALID_VALUES]', where VALID_VALUES is a list of the valid string values.

SYSTEM_SPECIFICATION := {'src_root': PATH,
                         'components': [COMPONENT_SPECIFICATION, ...],
			 'symbols': {SYMBOL_VALUE_PAIRS}
                        }
COMPONENT_SPECIFICATION := {'name': STRING,
                            'major_number': STRING,
                            'minor_number': STRING,
                            'patch_level': STRING,
                            'files': [FILE_SPECIFICATION, ...]
                           }
FILE_SPECIFICATION := {'from': PATH,
                       'to': PATH | 'to-file': PATH,
                       'os': STRING('win32'|'linux2'),
                       'chmod': CHMOD_OCTAL_MODE
                      }

The non-terminal PATH represents a Python literal string containing an absolute or relative file path using '/' as separator. You may also use environment variables in PATH expressions using the $VAR convention. The attribute 'src_root' denotes the root directory of all relative paths specified for the component files. If 'src_root' is relative it is relative to the current location of the sas-file.

The value of the 'from' key specifies from where the file is to be copied. The value of the 'to' key specifies the directory (relative to the root of the system assembly) to which the file is to be copied. The value of the 'to-file' specifies the file path (relative to the root of the system assembly) , including the file name, to which the file is to be copied. The keys 'to' and 'to-file' are mutually exclusive.

The key 'os' is optional and if present indicates that the specified file is only to be copied on the given os.

The key 'chmod' is optional and specifies the file permissions of the copied file. The non-terminal CHMOD_OCTAL_MODE is a Python literal string containing a chmod octal mode and is only applied on Linux. It is ignored on Windows. Important note: For historical reasons; if the key 'chmod' is not specified the following permissions (chmod octal mode) are set on Linux for the file: 750 if the file name suffix is '.py', '.spy' or '.so' (the file is thus considered an executable). For all other files the permissions are set to 640.

Using symbols

Unix shell style symbols in the form $SYMBOL or ${SYMBOL} can be used to specify file and directory path across the specification.

The values of the symbols are resolved, first looking up the symbol in the dictionary 'symbols' in the assembly specification and then looking up in the process environment.

E.g: $PREFIX/abc/${SYS_BUILD}dir-${VER}/bin/$MYFILEconf Includes 4 symbols PREFIX, SYS_BUILD, VER and MYFILEconf

An example

Here is a simple example of a sas-file for a fictive system 'Foo':

{'src_root': '../..'
 'symbols': {'rel': 'src/rel'}
 'components': [
 
    ### Client part of Foo ###
    {'name': 'Client',
     'major_number': '1',
     'minor_number': '2',
     'patch_level': '8',
     'files': [
        {'from': 'foo-client.exe', 'to': 'bin', 'os': 'win32'},
        {'from': 'foo-client', 'to': 'bin', 'os': 'linux2'},
        {'from': '$rel/readme.txt', 'to': ''},
        {'from': '$rel/license.txt', 'to': ''}
        ]
    },
       
    ### Server part of Foo ###
    {'name': 'Server',
     'major_number': '1',
     'minor_number': '4',
     'patch_level': '45',
     'files': [
        {'from': 'foo-server.dll', 'to': 'bin', 'os': 'win32'},
        {'from': 'foo-server.so', 'to': 'lib', 'os': 'linux2'},
        {'from': '$FOO_DEV/data/empty.db', 'to': '/data'}
        ]
    }
]}

Based on the above sas-file the System Assembler will generate a directory structure like this on Windows (assuming all files are found!):

+ Foo-1.2.45
  + Client-1.2.8
    + bin
      - foo-client.exe
    - readme.txt
    - license.txt
  + Server-1.4.45
    + bin
      - foo-server.dll
    + data
      - empty.db

Command line usage

The SystemAssembler can be invoked from the command as a normal Python script. Here are the command line options:

$ ./SystemAssembler.py --help
Usage: SystemAssembler.py -fFILE -tDIR [-cCOMPONENT] [-h]
  -f|--file FILE                File containing the system assembly specification.
  -t|--target_dir DIR           Target directory to assemble system in.
  -c|--component  COMPONENT     Component in system to assemble, default is all.
  -h|--help                     Prints this message.

Using SystemAssembler in SCons

Drop the file 'SystemAssembler.py' in /usr/lib/scons/SCons/Tool'.

Assuming you have a .sas file called 'foo.sas' you can invoke the SystemAssembler in your SConstruct file like this:

import os
# Create a SCons environment with the SystemAssembler tool
env = Environment(ENV = os.environ, tools = ['default', 'SystemAssembler'])
env.Append (arguments = ARGUMENTS)

# Since the SystemAssembler tool wants a directory as its target we need to 
# make sure we explicitly convert the directory name to a SCons directory node.
# Otherwise SCons will try to convert the target name to a SCons file node,
# which will fail.

system_target = env.Dir ("Foo-0.1")
env.SystemAssemble (system_target, "foo.sas")

A note on implementation

The SystemAssembler tool is registered as a SCons builder tool and it also sets up a scanner for .sas files. The scanner reports dependencies on all files listed in the .sas file to SCons so it can keep track of dependencies.

The SystemAssembler tool also sets up an emitter. The emitter complements the target directory with a target file called 'SystemAssembler.log' which will be created in the current working directory. This is needed because SCons always consider a directory, if it exists, as up-to-date. If we only have a directory as a target, it will never be rebuilt, even if a file it depends on changes. By adding the normal file 'SystemAssembler.log' to the list of targets, SCons will rebuild all the targets if any of the dependent files are updated.

Todo and improvement suggestions

Highest priority:

  • Allow a single top level component so we can write:
    {'symbols':  {'PROJPATH':'../..'},
     'name': 'foobar',
     'files': [
            {'from': '$PROJPATH/src/foobar', 'to': './var/www/foobar'},
             ...
       ]
    
    Rationale: It's allows for simpler sas-files and means we can avoid an extra directory level in the resulting directory layout (for single component systems!).
  • Remove requirement on specifying 'src_root'. Rationale: It removes a special key. The current rationale for having this key is that it enables one to define a default root path for all directory paths.
  • Add functionality for textreplacement, eg:
            {'from': '$PROJPATH/ins/deb/control', 'to-file': 'DEBIAN/control', 'replace': {"{{VERSION}}": "$VERSION", "{{PACKAGE_VERSION}}": "$PACKAGE_VERSION"}},
    
    where $VERSION and $PACKAGE_VERSION have been defined in 'symbols' like this:
    'symbols':  {'VERSION': '0.1', 'PACKAGE_VERSION': '1'},
    
    This means that the SystemAssembler will replace all occurrences of the string '{{VERSION}}' in the file 'DEBIAN/control' with the value of the symbol '$VERSION'. For a Debian control file and the symbols above that would mean:
    Package: foobar
    Version: {{VERSION}}-{{PACKAGE_VERSION}}
    
    being replaced with:
    Package: foobar
    Version: 0.1-1
    
    Rationale: Avoid having to implement text replacement stuff in the SConstruct file!
  • Allow for symbols to be passed to the SystemAssembler. Both via the SCons tool interface and via the command line. A SConstruct file could look like this:
    symbols = {'VERSION': '0.1', 'PACKAGE_VERSION': '1'}
    targets = env.SystemAssemble (system_target, "sil-dashboard.sas", symbols)
    
    Rationale: This (together with the text replacement mechanism above) would allow one to specify version information in one place in the SConstruct file and have it affect all desired files and directories!
  • Ensure it works fine as standalone command line tool. Suggested name: 'sysasm'! Rationale: It would be nice and may lead to the tool being used for other purposes.
  • Create a Debian package for the SystemAssembler! Install both as standalone command line program and SCons tool. Rationale: Simpler and easier installation!
  • Generate warning if a given file is not under version control (check for svn and git!). Rationale: A small but useful QA feature!
  • Enable for copying empty directories, in effect for creating empty directories. Rationale: This is useful when generating file hierarchy layouts for Debian packaging. Eg:
    {'from-dir': '$SRC/ins/deb/var/log/foo', 'to-dir': 'var/log/foo', 'os': 'linux'},
    

Lower priority:

  • Make sure SystemAssembler is registered as a default SCons tool, so it doesn't have to be registered in the !SConstruct file.
  • Provide an include mechanism for including other sas-files. Note: This is questionable because one of the main purposes of SystemAssembler is that a .sas file should give full picture of all files belonging to a system. However if there was a command line switch --full-listing that could output a complete listing of specified files including those specified in included sas-files, this could be useful feature.