Bootstrapping Jam

We start with an exchange between Toon Knapen and Dave Abrahams:

Toon:

Overall, I find the build-process for jam itself very fragile. For instance, when compiling the perforce jam on all our platforms, I always needed to start from a fresh copy for every platform. If for instance I tried to compile the same tree first on HP and then on IBM, it would not work.

Dave:

I am no make expert, but here's my impression of the bootstrap philosophy:

Make is really non-portable, so do something REALLY TRIVIAL with make in order to build jam0. Then do something more reasonable with jam0 to build jam. So, the make process just puts the object files in the same directory as the source. That is part of the reason you'd find yourself unable to rebuild using the same source tree: the .o files lying around are inappropriate for your new platform.

I don't think this is so bad: in theory, once you have a Jam executable on a platform, you can rebuild at any time by just running jam in the source directory, which theoretically causes objects to be generated in a subdirectory based on the platform (e.g. bin.ntx86 for my win2k box).

Toon:

(I do not know much about yacc, parsing etc. thus that certainly contributes to my problem ; on the other hand I understand that bootstrapping a build-tool is complicated).

The dependency on YACC scares me though. I expect that in the future I will be trying to use the build systems on e.g. NEC SX (vector-parallel machines). But if I understand correctly, yacc is not really necessary, previous yacc output can be reused ?

Dave:

That's right. In fact, the Windows build is set up not to use yacc by default. There is a section in Jambase which defines global variables for a variety of platforms; the YACC variable is among them. We can disable yacc for any platform which doesn't have yacc or generate reliable parsers, since the generated source files are bundled with the Jam sources. The only side-effect is that you won't be able to change the Jam grammar when developing Jam on these platforms.

Here is more detailed information from David Turner, the author of FTJam, on which Boost Jam is based:

I'll try to detail the Jam (and FTJam) build process a bit here to help you understand what's probably going wrong.

  some C source files used during the build process are automatically
  generated from other ones. More specifically:

    - "jambase.c" is generated from "Jambase" by running the program
      named "mkjambase". both files contain the default Jam rules to
      be compiled within the executable..

    - "mkjambase" is itself generated by compiling the stand-alone
      program source named "mkjambase.c"

    - "jamgram.c" is generated by "jamgram.yy" which is processed
      by the "yyacc" shell script on Unix. Note that this script also
      generates a "jamgram.y" during its execution..

  when you unpack the source package to compile it, the auto-generated
  files are already present in the current directory.

  when you invoke "make" to build Jam, the Makefile first creates an
  executable named "jam0" in the current directory, by using all versions
  of the auto-generated files that are already available. When completed,
  "jam0" is invoked directly..

  the later is a preliminary jam executable that will uses the rules
  contained in the "Jamfile", as well as the defaults that were contained
  in the original "jambase.c". These rules do the following:

    - they create a directory, like "bin.ntx86", where further objects
      and executables will be placed. let's call it the "target" directory

    - it generates the "mkjambase" program in the target directory,
      then invoke it to re-generate "jambase.c" from "Jambase"

    - on Unix (and only on Unix), it also re-generates "jamgram.c"

    - then, it recompiles the jam executable within the target
      directory..

  Now, it seems that a special compilation flag is needed on HP-UX.
  If you want to modify the jam build process to support it correctly,
  you need to:

     1. Modify the Makefile to add the "-Ae" to the CFLAGS definition,
        _and_ to the default "jambase.c". This will let you build
        a _working_ "jam0" correctly.

     2. Modify the "Jambase" file to support the new flag (i.e. in the CCFLAGS
        definition on HP-UX systems). This will ensure that the final jam
        executable will compile with correct defaults..

  another quick way to do the same thing, but with less file modifications:

      export CFLAGS=-Ae                     ==> used by the Makefile
      export CCFLAGS=-Ae                    ==> used by the Jamfile/Jambase?
      ... modify only the "Jambase" file ...
      make

  and you should get a working "jam" executable in "bin.xxxx"

  I have also just updated the "Jambase", "jambase.c" and "Makefile"
  in the current FTJam sources. Could you test it to see if it solves
  your problem (you should be able to grab a copy through the Perforce
  public depot, or simply ask me for a .zip or .tar.gz package file).