How to write independent test suites with nextest

[Nugine]

I'd like to test different web services which use the same application-level protocol.

So I plan to write a reusable e2e test suite.

• distributes a single binary independent from Rust toolchain
• inputs configurations (host:port, test filters, ...)
• outputs test summary and results
• runs with awesome features from nextest

It will be like a custom test harness with nextest runner, packed in the same binary.

My questions:

• Is it possible?
• How to implement?

[sunshowers]

Thanks for the great question! This is honestly not a use case I'd thought about before.

I think the use case makes sense but I'm worried it's a lot of work. Currently, cargo-nextest (including nextest-runner) is really tied to Cargo and rustc. It uses Cargo to discover binaries, to figure out what environment variables to set, and for a number of other things.

This is not inherent to nextest's model: you could certainly imagine the runner working against a more abstract test discovery and running interface. I think doing so is probably going to make nextest's code better as well, and also expose a bunch of hidden dependencies to Cargo. But in order to achieve this you'd probably need to roll up your sleeves and do the design and implementation work.

This would also likely enable nextest to work against non-Cargo build systems like Buck and Bazel, something that's been desired for a while: #1790. And also likely against non-Rust languages as well. It essentially decouples nextest's features from their current reliance on Cargo and rustc.

To set expectations: I don't anticipate getting to this any time soon. Maybe in a world where we start using Buck2 at my employer, but not any time soon. But I think it would be a good idea if you'd like to try and flesh out a design for this using traits.

[Nugine]

Basically what I'm looking for is a full-featured embedded test framework (custom harness + runner).
It's like gtest and catch2 in C++ world. I think there is an empty place in Rust ecosystem and nextest seems to be the closest.