Suggestion: Documentation tests

[Tim-Evans-Seequent]

With Python and Rust I can put examples into my doc-comments and run them as tests via pytest or cargo test respectively. Is there any chance of Doxide helping to extract these examples from the doc-comments and generate unit-test files from them?

This would obviously be more complex for C++ because nobody can agree on a single unit test framework to use. Having a configurable header and footer for the output files, and for each test case, might be enough.

You'd also probably want something like Rust has where only some lines of the code block are shown in documentation, but all of them are output to the test file. Rust uses a # prefix on lines not included in the docs, but that would be awkward for C++.

#include directives would need to be extracted and moved to the top of the output test file.

[lawmurray]

Thanks for the suggestion @Tim-Evans-Seequent . I think I'm seeing two aspects here:

1. Showing code examples/tests in the documentation that can be run. The "Run this code" buttons on cppreference.com are backed by the Coliru online compiler, so that might be an approach.
2. Extracting those examples/tests from the doc comments to build into unit tests. Could be a heavy lift, and risk of inventing yet another unit test framework for C++, so I'm less enthusiastic about this one, but also... why not? If you or someone else has the bandwidth to put together a compelling design and implementation then happy to take contributions. The complexity, to my mind, is in building and running the tests given the user will have a preferred build system. But perhaps just extracting the tests to source files is enough to start with, and the user can integrate into their build system as they please from there.

[Tim-Evans-Seequent]

I hadn't even thought about the first case, but I can see the value in it now, especially if users can also edit that code. That's getting into "Jupyter Notebook but C++" territory and could be a deep rabbit hole.

For 2 I think the one absolutely necessary feature would be some way of hiding lines of code in the documentation. With Rustdoc for example lines starting with # are executed as part of the test but not shown in the final docs. That character would be confusing in C++ so maybe @ would be better. That would let me write example code as Catch2 (or anything else) unit tests like so:

/// Calculates factorials
///
/// ```c++
/// @ #include <catch2/catch_test_macros.hpp>
/// @ TEST_CASE("Factorial example", "[doctest]") {
///     REQUIRE(factorial( 1) == 1);
///     REQUIRE(factorial( 2) == 2);
///     REQUIRE(factorial( 3) == 6);
///     REQUIRE(factorial(10) == 3'628'800);
/// @ }
/// ```
uint32_t factorial( uint32_t number ) {
    return number <= 1 ? number : factorial(number-1) * number;
}

The docs would include only the REQUIRE lines and discard the common indentation as well.

Actually extracting the example code and transforming it into a runnable unit test could be done by a separate tool. Doxide could help with by making it easier to get metadata like what class/function is being documented for example, but it isn't necessary. Even once that information is extracted it could just be passed to a script to process, making it independent of the testing framework used.

I'm not sure if I will have time to work on this, but I'll post a patch if I end up writing it. I haven't found any other good tools for doing this yet.