How to benchmark Khepri releases & development branch?

[dumbbell]

I would like to benchmark Khepri against Mnesia and Khepri releases among themselves and against the main development branch. I would love to publish them automatically as a GitHub page for instance.

Any advice on which tool to use to do that?

I'm considering Benchee, but that alone won't work because a Rebar3- or Mix-based project can't depend on several versions of the same dependency. Therefore, I would need to aggregate the results of indedepent project (one per Khepri version).

[dumbbell]

I just discovered Benchee can save results and load them into a later run to compare two branches. Perhaps I can design a benchmark where we can switch between Mnesia and Khepri and configure the version of Khepri. This way, we run the same benchmark multiple times: once with Mnesia, the rest with a specific release of Khepri, then a last run with the main branch of Khepri, loading the results of all previous runs to prepare the final report.

[dumbbell]

I prepared a micro-benchmarking tool, based on erlperf, to compare Khepri and Mnesia.

The tool is available in a separate repository: https://github.com/rabbitmq/khepri-benchmark.

It compares a disk-copy table in Mnesia (with its default configuration) to Khepri using both safe settings (default) and unsafe settings (fsync(2) disabled). The benchmark tries insertions and deletes with concurrency gradually increasing from 1 to 200 parallel workers. The same benchmark is executed with a single node and a 3-node cluster (running on the same host though).

The benchmark runs automatically in GitHub Actions and an HTML report is published to https://rabbitmq.github.io/khepri-benchmark/. The performance in GitHub Actions does not reflect what we get with more CPU cores though.

[dumbbell]

Note that the current framework doesn't compare different versions of Khepri and/or Ra. This would be nice to have, but this tool is a good start. Another thing missing from this benchmark is that it doesn't show the latency.

[dch]

@dumbbell these numbers look really good already! And a great choice of OS for the benchmarking :)

Here's some more from a bunch of systems I have available https://gist.github.com/dch/54b859f97fef263c2fe49bd5c1070d2a

I will re-do these once I have an OTP25 RC working for the arm64 JIT comparison. Let me know if you want any more info.

[dumbbell]

Thank you very much Dave!

And a great choice of OS for the benchmarking :)

An obvious one ;-)

Your results are interesting. I will analyze them on Monday with @kjnilsson. We will probably have a few questions on the hardware for things that are not showed in the system information.