Bringing B2-Style Test Granularity to CMake

By Alan Freitas on
Jul 10, 2025

Introduction

Boost libraries typically maintain granular unit tests using Boost.Build (B2). B2 provides a run rule that makes it easy to define many independent test targets from a single source file or executable. Each test case can be listed, invoked, and reported separately, which improves developer workflow, test clarity, and CI diagnostics.

However, Boost’s CMake integration has lacked this granularity. When Boost libraries are built with CMake, the typical approach is to define a single test executable and add all test suites as a single test in CTest with add_test(). As a result, when running tests with CTest, developers lose the ability to see individual test failures in isolation, run only subsets of tests, or leverage parallel execution at the test level.

The goal of this work is to bridge that gap. We want to replicate the B2 “one executable, many independent tests” idiom in CMake. Specifically, we want to use modern CMake techniques to split a single unit test executable into multiple independent CTest targets, while preserving the flexibility and simplicity of the Boost testing style.

Problem Analysis

To understand why splitting tests into independent CTest targets is non-trivial, it helps to look at how CMake’s build and test model is structured.

When building and testing libraries with CMake, the developer usually has a workflow of four key phases:

  • Configuration step: This is where CMakeLists.txt files are processed and commands like add_executable and add_test() are called. Test targets must be defined here, so CTest knows about them.
  • Build step: This is when the underlying build system (the CMake “generator”: e.g., Ninja or Make) compiles sources and produces executables, including unit test binaries.
  • Test step: This is when ctest runs the defined tests, using the executables built in the previous step.
  • Installation step: This is where the built libraries and executables are installed to their final locations. This step is not directly relevant to the problem at hand, but it’s part of the overall CMake workflow.

At the configuration step, we would like to have something like

add_test(NAME test_a COMMAND my_unit_test_executable a)
add_test(NAME test_b COMMAND my_unit_test_executable b)
add_test(NAME test_c COMMAND my_unit_test_executable c)

instead of

add_test(NAME my_unit_test_executable COMMAND my_unit_test_executable)

The fundamental obstacle is that, in the general case, you cannot know what tests exist inside a unit test executable until you can run it. Many modern test frameworks (like Boost.Test, Catch2, GoogleTest) support listing their available tests by running the executable with a special argument (e.g., --list-tests). But this only works after the executable is built.

In other words:

You need the executable to discover the tests, but CMake requires you to declare the tests in the configuration phase before building the executable in the build step.

This dependency cycle is the core problem that makes it difficult to reproduce B2’s run rule semantics in CMake. Without special handling, you’re forced to treat the entire unit test binary as a single test, losing the ability to register its internal test cases as independent CTest targets.

The solution to this problem involves the TEST_INCLUDE_FILES directory property, which allows you to specify additional files that CTest should consider when running tests. By leveraging this property, we can dynamically generate a CMake script that defines individual add_test() calls for each test case found in the unit test executable.

So we can use

set_property(DIRECTORY
    APPEND PROPERTY TEST_INCLUDE_FILES "${TEST_SUITE_CTEST_INCLUDE_FILE}"
)

to include a generated CMake script that contains the individual test definitions. This allows us to run the executable post-build, extract the test names, and then register them with CTest in a way that mimics the B2 experience. Other modern test frameworks have explored this feature to provide an automated test discovery mechanism for their libraries in CMake. For example, Catch2’s catch_discover_tests() and GoogleTest’s gtest_discover_tests() run the built test executable with a listing flag (like --list-tests) to extract individual test cases and generate separate add_test() entries for each.

Design Overview

Since CMake requires test registration in the configuration step, but we can only discover test cases after building the executable, we introduce an approach to bridge that gap. The high-level plan is:

  • Build the executable: Compile the unit test executable as usual. The target is defined in the configuration step and built in the build step.
  • Post-build step: After building, run the executable with --list-tests (or equivalent) to enumerate all available test cases. This is achieved with a custom command that runs after the build completes.
  • Generate a CMake script: This post-build step writes a .cmake file containing one add_test() call for each discovered test case.
  • Conditional inclusion: The main CMake configuration includes this generated script only if it exists, so the tests appear in CTest after they’re generated. The new script is included using the TEST_INCLUDE_FILES property, which allows CTest to pick it up automatically.

This approach effectively moves test discovery to the build phase while still registering the resulting tests with CTest in the configuration phase for subsequent runs.

This process is transparent to the user. In Boost.URL, where we implemented the functionality, the test registration process went from:

add_test(NAME boost_url_unit_tests COMMAND boost_url_unit_tests)

to

boost_url_test_suite_discover_tests(boost_url_unit_tests)

Implementation Details

This section describes the approach bottom-up, showing the overall mechanism of discovering and registering independent test targets in CMake.

The Test Listing Extractor Script

The first piece is a small CMake script that runs the compiled test executable with --list-tests (or an equivalent flag your test framework supports). It captures the output, which is expected to be a plain list of test case names.

For example, suppose your unit test executable outputs:

UnitA.TestAlpha
UnitA.TestBeta
UnitB.TestGamma

The script saves these names so they can be transformed into separate CTest targets.

Example command in CMake:

execute_process(
        COMMAND "${TEST_SUITE_TEST_EXECUTABLE}" ${TEST_SUITE_TEST_SPEC} --list-tests
        OUTPUT_VARIABLE TEST_SUITE_LIST_TESTS_OUTPUT
        ERROR_VARIABLE TEST_SUITE_LIST_TESTS_OUTPUT
        RESULT_VARIABLE TEST_SUITE_RESULT
        WORKING_DIRECTORY "${TEST_SUITE_TEST_WORKING_DIR}"
)

Generator of CMake Test Definitions

Once the list of tests is available, the script generates a new .cmake file containing one add_test() call per discovered test. This file effectively defines the independent CTest targets.

Example generated tests.cmake content:

add_test(NAME UnitA.TestAlpha COMMAND my_test_executable UnitA.TestAlpha)
add_test(NAME UnitA.TestBeta COMMAND my_test_executable UnitA.TestBeta)
add_test(NAME UnitB.TestGamma COMMAND my_test_executable UnitB.TestGamma)

This approach ensures each test is addressable, selectable, and independently reported by CTest.

Post-Build Step Integration

CMake can’t know these test names at configuration time, so we hook the test listing step to the build phase using a POST_BUILD custom command. After the test executable is built, this command runs the extractor and generates the script file defining the tests.

Example:

add_custom_command(
        # The executable target with the unit tests
        TARGET ${TARGET}
        POST_BUILD
        BYPRODUCTS "${TEST_SUITE_CTEST_TESTS_FILE}"
        # Run the CMake script to discover tests after the build step
        COMMAND "${CMAKE_COMMAND}"
        # Arguments to the script
        -D "TEST_TARGET=${TARGET}"
        -D "TEST_EXECUTABLE=$<TARGET_FILE:${TARGET}>"
        -D "TEST_WORKING_DIR=${TEST_SUITE_WORKING_DIRECTORY}"
        # ...
        # The output file where the test definitions will be written
        -D "CTEST_FILE=${TEST_SUITE_CTEST_TESTS_FILE}"
        # The script that generates the test definitions
        -P "${TEST_SUITE_DISCOVER_AND_WRITE_TESTS_SCRIPT}"
        VERBATIM
)

This ensures the test listing happens automatically as part of the build.

Including Generated Tests

The main CMake configuration includes the generated .cmake file, but only if it exists. This avoids errors on a test pass before the executable is built. This could happen because the user is calling ctest before the build step completes, because the test executable was not built, or because the cache was invalidated.

So the discovery function uses the example pattern:

if(EXISTS "${CMAKE_BINARY_DIR}/generated/tests.cmake")
    include("${CMAKE_BINARY_DIR}/generated/tests.cmake")
endif()

And this is the file that the test step will ultimately include in the CTest run, allowing CTest to see all the individual test targets.

CMake Function for Reuse

To make this easy for other libraries, the pattern can be wrapped in a CMake function. This function:

  • Defines the POST_BUILD rule for the given target.
  • Encapsulates the details of running the extractor script.
  • Ensures consistent output locations for the generated test definitions.

Example usage:

boost_url_test_suite_discover_tests(boost_url_unit_tests)

This approach lets library maintainers adopt the system with minimal changes to their existing CMake setup, while maintaining Boost’s fine-grained, many-target test philosophy.

When we look at CI results for Boost.URL, this is the only thing we used to have:

  /__w/_tool/cmake/3.20.0/x64/bin/ctest --test-dir /__w/url/boost-root/build_cmake --parallel 4 --no-tests=error --progress --output-on-failure
  Internal ctest changing into directory: /__w/url/boost-root/build_cmake
  Test project /__w/url/boost-root/build_cmake
      Start 1: boost_url_unit_tests
      Start 2: boost_url_extra
      Start 3: boost_url_limits
  1/3 Test #2: boost_url_extra ..................   Passed    0.00 sec
  2/3 Test #3: boost_url_limits .................   Passed    0.00 sec
  3/3 Test #1: boost_url_unit_tests .............   Passed    0.02 sec
  
  100% tests passed, 0 tests failed out of 3
  
  Total Test time (real) =   0.02 sec

And now we see one unit test per test case:

  /__w/_tool/cmake/3.20.0/x64/bin/ctest --test-dir /__w/url/boost-root/build_cmake --parallel 4 --no-tests=error --progress --output-on-failure
  Internal ctest changing into directory: /__w/url/boost-root/build_cmake
  Test project /__w/url/boost-root/build_cmake
        Start  1: boost.url.absolute_uri_rule
        Start  2: boost.url.authority_rule
        Start  3: boost.url.authority_view
        Start  4: boost.url.compat.ada
   1/76 Test  #1: boost.url.absolute_uri_rule ..........   Passed    0.01 sec
        Start  5: boost.url.decode_view
   2/76 Test  #2: boost.url.authority_rule .............   Passed    0.01 sec
        Start  6: boost.url.doc.3_urls
   3/76 Test  #3: boost.url.authority_view .............   Passed    0.01 sec
        Start  7: boost.url.doc.grammar
   4/76 Test  #5: boost.url.decode_view ................   Passed    0.01 sec
        Start  8: boost.url.encode
   5/76 Test  #4: boost.url.compat.ada .................   Passed    0.01 sec
        Start  9: boost.url.error
   6/76 Test  #6: boost.url.doc.3_urls .................   Passed    0.01 sec
        Start 10: boost.url.format
   7/76 Test  #7: boost.url.doc.grammar ................   Passed    0.01 sec
        Start 11: boost.url.gen_delim_chars
   8/76 Test  #8: boost.url.encode .....................   Passed    0.01 sec
        Start 12: boost.url.grammar.alnum_chars   
  ...

meaning that each test is now executed and reported individually and in parallel, allowing developers to see which specific tests passed or failed, and enabling more granular control over test execution.

Conclusion

This approach brings fine-grained tests into the modern CMake Boost workflow. By splitting a single test executable into multiple independent CTest targets, maintainers gain:

  • More granular failure reporting: CI logs show exactly which test case failed.
  • Better developer experience: Developers can run or re-run individual tests easily.
  • Improved parallel execution: Faster test runs in CI and locally.
  • Better IDE integration: IDEs can show individual test cases.

For other Boost libraries considering adopting this pattern, the only requirement is that their test executables support a --list-tests (or equivalent) command that outputs the available test cases. Once that’s available, the necessary CMake changes to define an equivalent function are minimal:

  • Add a POST_BUILD step that runs the listing command and generates the .cmake file.
  • Conditionally include that generated file in the main CMakeLists.

If the output of --list-tests is one test suite per line, the existing script can be used as-is. This small investment pays off with a much more maintainable and CI-friendly testing setup. I encourage other maintainers and contributors to try this technique, refine it, and share feedback.

The complete script and CMake snippets are available in the Boost.URL repository at commit #a1a5d18.

All Posts by This Author