Rigorous automated testing

[JackKelly]

Ideas from Jon Gjenset's talk Towards Impeccable Rust at Rust Nation UK 2024 on Wednesday 27th March 2024:

Automatically check for memory leaks and UB

• Miri: "An experimental interpreter for Rust's mid-level intermediate representation (MIR). It can run binaries and test suites of cargo projects and detect certain classes of undefined behavior,"
• sanitisers (like thread sanitisers)

Chaos testing

"Anything that can go wrong will go wrong (and at the worst possible time)."

Useful crates:

• turmoil (for async code: change the order in which Futures are polled)
• shuttle (for sync code)
• Chaos testing input values: quickcheck or proptest. Try random inputs, and ensure the code still behaves(!)
• cargo-mutants: logic chaos. Alters your code (e.g. changing x+1 to x-1) and ensures tests fail.

Exhaustive testing

Useful crates:

• Loom: Tries all possible and distinguishable concurrent executions. Tries all ways that threads could interleave execution order. This testing can take hours to run. It only works well for relatively simple systems (otherwise the number of permutations explodes).
• Kani: Analyses your code to find the specific values that exercise many different conditional branches. e.g. if you have if x == 4000 { foo() } else { bar() } then Kani will figure out that the only two important values for x are 4000 and anything other than 4000.

Related

• Rigorous benchmarking #119
• Document decisions taken #120