The obuild program

obuild is a "builder builder".

It permits, from an XML description of a package, to build shell scripts to build the package. Then shell scripts to compile the source code, build libraries, plugins, programs and also to help building a binary release in case of a "project" defined as a set of packages.

A simple example

Let us suppose that you have a package "Xxx" with the structure :

   Xxx/v1r0/include/file.h
   Xxx/v1r0/source/file.cpp
   Xxx/v1r0/application/main.cpp

and that you want to build a library libXxx having the code of file.cpp in it, and also to build an application named "Xxx_program" having its main in main.cpp that uses code defined file.h and file.cpp.

Then you have to create a obuild directory with a this.obuild file :

   Xxx/v1r0/obuild/this.obuild
 having the content :
   <obuild>
     <package name="Xxx" version="v1r0">

      <library name="Xxx">
        <files>../source/*.cpp</files>
        <cxx_include>${obuild_Xxx_path}/include</cxx_include>
      </library>
  
      <application name="Xxx_program">
        <file>../applications/main.cpp</file>
        <cxx_include>${obuild_Xxx_path}/include</cxx_include>
        <lib>Xxx</lib>
      </application>

      <!-- Run time environment : -->
      <setup>  
        <load_library_path/>
      </setup> 

     </package>
   </obuild>

Then from the obuild directory, you run the obuild program :

   OS> obuild

that, by reading informations given in the this.obuild file, will create the various material to build the package (and then here to build the libXxx library and the Xxx_program application).

When done, you build the Xxx package with :

   UNIX> sh/build
    DOS> DOS\build

The build scripts create a parallel bin_obuild directory in which the libraries, plugins, applications are build.

Then to run the Xxx_program application do :

    csh> source setup.csh
   ( sh> . ./setup.sh )
  ( DOS> call setup.bat )
 and :
  shell> ../bin_obuild/Xxx_main
  ( DOS> ..\bin_obuild\Xxx_main.exe )

Sourcing the adequate "setup" script permits to set the adequate path variable to find at run time the libXxx shared library. (For a library, by default the script build_lib_Xxx attempts to build a shared one).

Produced scripts

obuild produces under the sh directory the shell scripts to build components for UNIXes and under the DOS directory the ones to build on Windows from a DOS prompt.

For each component (library, plugin, application) named "Yyy" are created :

    sh/build_[lib,dll,app]_Yyy
    DOS\build_[lib,dll,app]_Yyy.bat

For example, in the case of the upper Xxx package, are created :

    sh/build_lib_Xxx
    DOS\build_lib_Xxx.bat
    sh/build_app_Xxx_program
    DOS\build_app_Xxx_program.bat

Some general scripts are also created like the global build scripts :

    sh/build
    DOS\build.bat

These can be used to build at once the components of a package. Options to the build (build.bat) general script are :

    UNIX> sh/build                # to build silently.
    UNIX> sh/build -v             # to have a verbose mode.
    UNIX> sh/build -g             # to build in debug mode.
    UNIX> sh/build -group <group> # to build only some specific target.
  ( UNIX> sh/build -v -group Zebra )
  ( UNIX> sh/build -v -group Python )
  ( UNIX> sh/build -v -g -group Qt )
 and on Windows :
     DOS> DOS\build                
     DOS> DOS\build -v
     etc...

Other (optional) general scripts are the setup scripts :

    setup.sh
    setup.csh
    setup.bat

They are intended (if created) to be "sourced" under the adequate shell to setup the package environment before any attempt to run any application of this package. A setup script may set the LD_LIBRARY_PATH under a Linux in order to find shared libraries at run time, the PATH variable under a Windows to find related DLLs, etc... In case of a package depending of other ones, a setup script may call the ones of the related packages if needed. The creation and tuning of the setup scripts is specified by the <setup> tag in the this.obuild file.

The XML tags. The obuild semantic

In the below schema we describe the tags hierarchies and after a description of them :

    obuild
      package
        use
          c_include cxx_include
          lib
        library
          file files directory
          c_include c_macro c_options
          cxx_include cxx_macro cxx_options
        dll
          file files directory
          c_include c_macro c_options
          cxx_include cxx_macro cxx_options
        application
          file files directory
          c_include c_macro c_options
          cxx_include cxx_macro cxx_options
          lib
        reference
        setup
          path env load_library_path run_path
        release
          copy_files remove_files copy_file

    obuild
      project
        reference
        zip

    obuild
      foreign
        native_version
        variable
        c_include
        cxx_include
        lib
        load_library_path
        check_directory
        release
          copy_files remove_files copy_file

<obuild>

It is the general tag that encloses other obuild tags.

<package>

It is the head tag to describe a package. Main children are the <library>, <dll>, <application>, <setup> tags.

Attributes of this tag are name and version. For example :

  <package name="Xxx" version="v1r0">

When executing obuild, the program checks that the given name matches the one in the file system tree directory :

    <some path>/Xxx/obuild

<library>

The tag to describe a library, archived or shared. Files to be compiled and put in the library are described with the <file>, <files>, <directory> tags.

<dll>

The tag to describe a dll (or plugin). Files to be compiled and put in the plugin are described with the <file>, <files>, <directory> children tags. (The <files> one is intended to pass wildcards).

<application>

The tag to describe an application. Files to be compiled and put in the application are described with the <file>, <files>, <directory> children tags. In particular one of the file must contain a main entry point.

Libraries to link the application with are specified with the <lib> children tag (which must not be confused with the <library> tag).

Using material of another package

Packages can be organised in a hierarchical way. An application or a dll of a package may be linked with a library defined in a sub or "used" package. This "usage" dependency is specified by having a <use> tag under the <package> one. To find packages obuild uses the OBUILD_PATH environment variable. OBUILD_PATH is a list of directories separated by colon on UNIXes and a dot-colon on Windows.

For example in the zip this.obuild file, can be found under the package tag :

    <use name="zlib">
      <c_include>include</c_include>
      <lib>osc_zlib</lib>
    </use>

and then under the zip_minizip application tag :

      <lib copy="lib" from="zlib"/>

When building the build_app_zip_minizip script, obuild will look for the lib tag of the <use> zlib tag to find the exact name of the lib to link with (here osc_zlib). Then, by using the OBUILD_PATH, it will put the code to find exactly the library to link with. (For example on UNIXes it will arrange so the the -L link option is valid).

Projects. Binary releases

We had the need to deliver group of packages on the form of binary releases. The <project> and <release> tags had been introduced for that. In OpenScientist we had found convenient to introduce special packages to handle projects and the related obuild material. For example we have the osc_batch package having the file system structure :

   osc_batch/obuild/this.obuild

The this.obuild of a project package has the form :

  <obuild>
    <project name="osc_batch" version="16.8">
      <reference package="obuild" file="obuild/this.obuild"/>
      <reference package="foreign" file="obuild/this.obuild"/>
      ...
    </project>
  </obuild>

Attributes of the <project> tag are name and version. The version is used in the name of the final zip file containing the binary release. Main children of the <project> tag are the references of the this.obuild of the various packages to be included in the project release (and then in the binary tar files). Applying obuild from the project obuild directory will create the :

    sh/build
    DOS\build.bat
 and :
    sh/build_release
    DOS\build_release.bat

The "build" script is used to build sequencially (as specified in the this.obuild) the packages of the project. When done you can use the "build_release" script to build the zip file containing the release. The zip file is put in the parallel bin_obuild directory of the project package.

The file system structure of a binary release is a common one :

    <project_name>/<version>/bin
    <project_name>/<version>/lib
    <project_name>/<version>/Resources

Under bin are put the applications of the various packages (then copied from the bin_obuild of the respective packages). Under bin are also put the dlls (plugins). Under lib are put the library to link with if building new application by using the binary kit. Under Resources are put, per packages, the include files needed to build new applications. For example the osc_batch kit contains the AIDA include files put under :

    osc_batch/<version>/Resources/AIDA/src/cpp/AIDA

The Resources directory contains also per package various other materials needed to operate a given application (for example Python scripts, .onx files, etc...)

By using the <release> tag under a <package> tag, you can can customize for a given package what goes in a binary release of a project containing this package. For example in the BatchLab this.obuild you can see :

    <release>
      <copy_dir>
        <from>scripts</from>
        <to>Resources/BatchLab/.</to>
      </copy_dir>
      <copy_dir>
        <from>examples</from>
        <to>Resources/BatchLab/.</to>
      </copy_dir>
      <copy_dir>
        <from>BatchLab</from>
        <to>Resources/BatchLab/.</to>
      </copy_dir>
    </release>

With the upper specifications, the obuild program will put in the build_release script of the osc_batch project, copy commands that will put in the osc_batch binary kit a copy of the scripts, examples and BatchLab (for include files) directories.

OpenScientist project packages

For OpenScientist we have the obuild, osc_batch and osc_vis project packages to build various kind of binary kits. The obuild permits to build binary kits containing only the obuild tool. The osc_batch permits to build kits containing no graphics and no GUI. They are intended to deliver light kits to help "capturing data" in the form of histograms and tuples to be put in files at various formats. The osc_vis permit to build kits containing the same than the osc_batch ones but with interactive applications like onxlab and opaw.

Handling foreign packages

By foreign packages we mean packages not handled by obuild. For example X11, Motif, Qt, Python, etc... It is assumed that a foreign package is installed by an installer of its own. The <foreign> tag, children of the <package> one, permits to describe how to find include files, libraries and other resources of a foreign package. In OpenScientist the package named "foreign" contains one .obuild file per foreign package. For example :

    foreign/pthread.obuild
    foreign/X11.obuild
    foreign/Qt.obuild
    foreign/Python.obuild
    ...

For example the X11.obuild looks like :

  <obuild>
    <foreign name="X11">
      <c_include platform="UNIX">/usr/X11R6/include</c_include>
      <lib platform="UNIX" 
         verbatim="yes">-L/usr/X11R6/lib -lXi -lXext -lXpm -lX11</lib>
    </foreign>
  </obuild>

If creating a new .obuild foreign file, the best is to look at the various .obuild file in the foreign package to have examples.

obuild -shell

If being in the obuild directory of a project package, then you can use the -shell option to enter the obuild shell and then enter a OS shell command to be executed on all the packages of the project. For example from the osc_vis package of a osc_vis_source kit :

     UNIX> cd osc_vis/obuild
 Set the OBUILD_PATH of the osc_vis project with :
      csh> source setup.csh
     ( sh> . setup.sh )
 Set PATH to run obuild (it assumes that obuild itself had been built).
      csh> source obuild-setup.csh
     ( sh> . ./obuild-setup.sh )
     UNIX> obuild -shell         
     obuild>> pwd

The upper pwd will be executed on all the packages of the project (here osc_vis). It will print the directory under :

     <package>

for all packages. This directory is the default working directory of the obuild shell for all packages.

The obuild shell is very convenient in case of customizing some foreign .obuild file and then be sure to apply on all packages. In this case on a UNIX do :

     obuild>> cd obuild;obuild
 and on Windows :
     obuild>> cd obuild && obuild

It will go in the obuild directory of each package and apply obuild.

Why obuild ?

Below are the coarse graining arguments that lead to the creation and usage of obuild.

XML

We wanted to describe a package by avoiding the introduction of a new syntax (and then a new parser) and XML looks natural for that. Some package being multi-drivers (for example OnX), we wanted to be able to split the description of a package in multiple files, for example one file per GUI in the case of OnX (Xt.obuild, Qt.obuild, etc...).

Standalone building scripts

One leading idea had been to arrange so that the builder-builder be not needed on common platforms to build and install a package from source. This in order to avoid to the installer the common "bootstrapping" problem of having to coope (and sometime debug) the installation of the builder-builder itself ! (Syndroma of a not working autoconf/configure script).

In principle, for common cases (like pure C++ standalone code), the material produced by obuild can be packed in the source distribution and reused to build the package without having to build and use the obuild tool itself on the target machine.

No implicit inheriting comportement

We wanted also something without automatic inheritance of compilation flags, link flags, etc... in case of a package using / depending of other packages. Then we wanted an explicit description of the material to build a component (library, plugin, program). Experience (with CMT) showed us that for a chain of packages with multi-drivers, an automatic inheritance of flags finished to be hard to trace and understand. Then we had preferred an explicit mechanism in this case. It may be more verbose, but at least it keeps the description more understandable.

DLL, plugin component

One point that finished to be painfull with CMT had been the lack of the dll or plugin keyword. For us a plugin is some executable code dynamically loadable. It is an "in between" between a shared library and a program. It has to be fully linked like a program but, like a library, does not contain a main. CMT has the library and application component keywords, but none for a plugin. And it appears that the building of a plugin cannot be reduced to the building of a shared library or a program. On some platform (MacOSX) someone have to use specific command to build a plugin (c++ -bundle on a MacOSX).

In obuild we have, beside <application> and <library>, the <dll> tag component that permits to deal with plugins.

Building binary release

We needed also a way to describe what should go in a binary release of a project (being defined as a set of packages). And the <release> tag permits us to do that.