Testing

Testing is an integral part of the Language.C project, and hopefully increases the confidence in the library.

Some preliminary test reports give an idea of what should be achieved.

Methodology

Wrapper scripts

The idea Duncan used to test the parser is to compile a project with CC set to a wrapper, which first calls the Language.C testing executable and then gcc. This way libraries can be tested by simply typing CC=cc-wrapper ./configure && make All tests are performed automatically, results are collected in streams of serialized haskell datatypes. Later on, results are rendered to HTML files.

Roundtrip testing

The testing executable first calls the C preprocessor (gcc -E) to preprocess the C file and then performs a set of different tests:

1. Timed parse ... Parses the given C file, times the parse and checks that no parse error occurred (performance test)
2. Pretty Print ... the AST to a new C file
3. Parse Pretty Printed ... file and check no parse error occured
4. Compare ASTs ... of parse and (parse . prettyPrint . parse)
5. (Maybe) Use gcc to compare to original file and the file resulting from (prettyPrint . parse)

For each test, failure / success as well as performance is recorded.

Other tests

To check that the AST actually contains everything parsed, one can use hand-crafted non-equivalence tests. Later on "does not compile" testing should also be supported.

Test Suites

Handcrafted

A few tests (smoke, associated with bugs) have been put into the test0 testsuite.

External

• The GCC project contains a large collection of C tests

gcc.c-torture gcc.dg The in-source comments are currently not processed.

• ruby snapshot

• Apache Portable Runtime

Test Facilities

Tests should be performed automatically, and should be simple to configure.

Currently, test are executed using bash scripts, collecting the results in a haskell datatype, plus some additional (plain text) report files. For example:

export C2HS_CC_LOGDIR=../testresults/ruby
export C2HS_CC_RESULT_FILE=../testresults/ruby.dat
export CC=ccw
cd ruby-src-0.8.3
./configure
make ruby

The result are automatically evaluated and rendered to an html file (like Pugs smoke tests).