Appendix E: CTest #

This appendix teaches CTest, the test driver that runs the compiled examples of this book. CTest is part of CMake. It discovers, runs, and reports the tests that a project registers. The book uses CTest to exercise every book_example target and to verify its output.

Registering a test #

A project enables testing with the enable_testing() command. The root CMakeLists.txt calls it before descending into the examples directory. A test is registered with add_test. The command names the test and gives the command to run:

add_test(NAME ch01_hello_run COMMAND ch01_hello)

The test ch01_hello_run runs the executable ch01_hello. CTest runs the executable in a separate process and reports whether it succeeded.

How book_example registers tests #

The book_example function in cmake/BookExample.cmake registers one test for every example. The relevant lines are:

add_test(NAME ${t}_run COMMAND ${t})
if(arg_EXPECT)
    set_tests_properties(${t}_run PROPERTIES
        PASS_REGULAR_EXPRESSION "${arg_EXPECT}")
endif()

The target name t comes from the file name. The test name appends _run to it. So book_example(ch01_hello.cpp EXPECT "hello, world") produces the test ch01_hello_run. The _run suffix keeps the test name distinct from the executable name.

Expected output verification #

A test passes when the program exits with status zero and its output satisfies the test properties. The property PASS_REGULAR_EXPRESSION holds a regular expression. When it is present, the test passes only if the program output contains a match for that expression.

A test fails when the program exits nonzero or when its output does not contain the expected regular expression. Both conditions are failures. The book relies on this rule to verify that an example prints what the text claims.

The EXPECT argument of book_example becomes the PASS_REGULAR_EXPRESSION. The book uses real regexes from the chapter files. A few examples:

book_example(ch01_hello.cpp EXPECT "hello, world")
book_example(ch02_quadratic.cpp EXPECT "roots: 2, 1")
book_example(ch03_point.cpp EXPECT "distance: 5.00")
book_example(ch03_parse_int.cpp EXPECT "value: 123|error at 2")

The last example shows a regex alternation. The vertical bar | means either alternative matches. The program prints value: 123 on success or error at 2 on failure, and the test accepts either.

Running the tests #

The book runs the tests through the preset:

ctest --test-dir build --output-on-failure

The flag --test-dir build points CTest at the build directory that the dev preset created. The flag --output-on-failure prints the full output of a failing test. Without it, CTest shows only a summary line for each test. The dev test preset sets this behavior in CMakePresets.json, so the flag and the preset agree.

Naming, labels, and verbosity #

Every test in this book is named after its example with a _run suffix. The naming makes a failing test easy to map back to its source file. The book does not use labels. A label groups tests under a name, and CTest can run only that group with ctest -L. The book has no need for grouping because every test is an independent example.

The default CTest output shows one line per test with a pass or fail result. The flag -V raises verbosity and prints each test command and its output. The flag -N lists the tests without running them. These flags help when you debug a single example.

Try this #

Run ctest --test-dir build -N and list the tests that the book registers. Pick one example that uses an EXPECT regex, such as ch03_parse_int. Temporarily change its regex in the chapter CMakeLists.txt to a string the program never prints, rebuild, and run ctest --test-dir build --output-on-failure. Confirm that the test fails because the output does not contain the expected expression, then restore the original regex.