[Home]Build Testing Framework

BOOST WIKI | RecentChanges | Preferences | Page List | Links List

Difference (from prior major revision) (no other diffs)

Removed: 1d0


Changed: 4,5c3,8
and begin using it ASAP. ''Vladimir, please
describe your testing system more completely here''.
and begin using it ASAP.





The Build.Build testing framework provides for setting up directory structure, invoking the build system on it, and examining results.

Changed: 7c10,14
His brief description:
Requirements. To use the testing framework, you'd need Python 2.1. Also, it should be possible to invoke Boost.Build by typing "jam" on the command line. This means that BOOST_BUILD_PATH and BOOST_BUILD_INSTALLATION should be set appropriately.

Basic operations. You start by invoking build_system_test.py script like this:
./build_system_test.py --toolset=gcc
It scans the current directory for zip files and expects each one to contains file "test.py" on the top level. That file describes tests (details below). Once tests contained in zip are known, for each test zip is extracted to a temporary directory, next test is run, and the directory is removed.

Changed: 9,20c16,21
Surely, there's a lot of disclaimers due -- it is very rought sketch, works
only with borland toolset, requires at least Python 2.1 (or maybe even 2.2).
However, features already present allowed me to write tests for all the
changes I've made.

Those features are:
# Unpacking a zip file to a temporary directory
# Touching or modifying content of any file.
# Running build systems and finding:
## commands run
## changes made (added/removed/modified/touched files)
# Examining changes made, e.g. saying: "I don't care for *.tds files, but build/bin/main/borland/debug/runtime-link-dynamic/main.obj should have appeared".
File "test.py" defines tests by defining classes derived from boost_build_system.tester and containing methods which names start with "test". For example:
class Sanity_test(boost_build_system.py):
def test(self):
self.rebuild()
self.expect_addition("build/bin/main/borland/debug/runtime-link-dynamic/main.exe")
It should be noted that test.py is not imported, but is "exec"d, so you need not write any import statements to access boost_build_system.tester (and import won't work now anyway :-) )

Changed: 22,23c23,44
There are no docs at the moment. In brief, code in boost_build_system.py
defines how build system is invoked, and build_test.py is the main program.


Here is the complete list of methods defined in boost_build_system.tester that are usefull in writing tests. Note that filenames always use "/" as separator, regardless of the current platform. They are lso relative to temporary directory into which the zip file was unpacked.
# set_content(self, name, new_content)
# touch(self, name)
# rebuild(self, subdir="", extra_options="", commands_only=0) -- runs jam in the specified subdir and passing it 'extra_options'. Sets self.command to all the commands that jam wanted to execute. Also sets internal attributes so that expect* functions work. If commands_only is 1, only list of commands is collected, but nothing is actually run
# run(self, exe): runs specified exe and sets self.status and self.output
The following methods examine difference made to the directory during last 'rebuild' invocation:
# expect_addition(self, names, dir="") 'names' can be a string or a list of string. 'dir' if given, specifies common prefix that is appended to all the names. If any of the files was not added, it is considered a test failure.
# ignore_addition(self, wildcard) -- names matching wildcard are ignored.
# expect_removal(self, names, dir="") -- similar to expect_addition
# expect_touch(self, names, dir="") -- similar to expect_addition
# expect_nothing_more(self) -- tests that all the changes are expected -- i.e. that for all changes either expect* method ignore_addition with matching wildcard was called.


Different toolset support.
At the moment, support for different toolsets in in inital state.
# Toolset specification. You should either set TOOLS environment variable to a toolset name, or explicitly give it using "--toolset=yfc" option.
# Using toolset information in tests.
## If test class defines class attribute "toolsets", which should be list of string, then the test will be run only for toolsets present in that list.
## Test can use global variable "toolset" in any way it likes.


Knows problems:
# import statement in tests does not work

Vladimir Prus has contributed a Python-based testing framework for the build system. I would like to check it in and begin using it ASAP.


The Build.Build testing framework provides for setting up directory structure, invoking the build system on it, and examining results.

Requirements. To use the testing framework, you'd need Python 2.1. Also, it should be possible to invoke Boost.Build by typing "jam" on the command line. This means that BOOST_BUILD_PATH and BOOST_BUILD_INSTALLATION should be set appropriately.

Basic operations. You start by invoking build_system_test.py script like this:

 ./build_system_test.py --toolset=gcc
It scans the current directory for zip files and expects each one to contains file "test.py" on the top level. That file describes tests (details below). Once tests contained in zip are known, for each test zip is extracted to a temporary directory, next test is run, and the directory is removed.

File "test.py" defines tests by defining classes derived from boost_build_system.tester and containing methods which names start with "test". For example:

 class Sanity_test(boost_build_system.py):
 def test(self):
  self.rebuild()
  self.expect_addition("build/bin/main/borland/debug/runtime-link-dynamic/main.exe")
It should be noted that test.py is not imported, but is "exec"d, so you need not write any import statements to access boost_build_system.tester (and import won't work now anyway :-) )


Here is the complete list of methods defined in boost_build_system.tester that are usefull in writing tests. Note that filenames always use "/" as separator, regardless of the current platform. They are lso relative to temporary directory into which the zip file was unpacked.
  1. set_content(self, name, new_content)
  2. touch(self, name)
  3. rebuild(self, subdir="", extra_options="", commands_only=0) -- runs jam in the specified subdir and passing it 'extra_options'. Sets self.command to all the commands that jam wanted to execute. Also sets internal attributes so that expect* functions work. If commands_only is 1, only list of commands is collected, but nothing is actually run
  4. run(self, exe): runs specified exe and sets self.status and self.output
The following methods examine difference made to the directory during last 'rebuild' invocation:
  1. expect_addition(self, names, dir="") 'names' can be a string or a list of string. 'dir' if given, specifies common prefix that is appended to all the names. If any of the files was not added, it is considered a test failure.
  2. ignore_addition(self, wildcard) -- names matching wildcard are ignored.
  3. expect_removal(self, names, dir="") -- similar to expect_addition
  4. expect_touch(self, names, dir="") -- similar to expect_addition
  5. expect_nothing_more(self) -- tests that all the changes are expected -- i.e. that for all changes either expect* method ignore_addition with matching wildcard was called.

Different toolset support. At the moment, support for different toolsets in in inital state.
  1. Toolset specification. You should either set TOOLS environment variable to a toolset name, or explicitly give it using "--toolset=yfc" option.
  2. Using toolset information in tests.
    1. If test class defines class attribute "toolsets", which should be list of string, then the test will be run only for toolsets present in that list.
    2. Test can use global variable "toolset" in any way it likes.

Knows problems:
  1. import statement in tests does not work

BOOST WIKI | RecentChanges | Preferences | Page List | Links List
Edit text of this page | View other revisions
Last edited November 6, 2001 8:24 am (diff)
Search:
Disclaimer: This site not officially maintained by Boost Developers