Skip to content

Latest commit

 

History

History
75 lines (58 loc) · 2.94 KB

BENCHMARKS.md

File metadata and controls

75 lines (58 loc) · 2.94 KB
Raw

Benchmarking Erlang/OTP

The Erlang/OTP source tree contains a number of benchmarks. The same framework is used to run these benchmarks as is used to run tests. Therefore in order to run benchmarks you have to release the tests just as you normally would.

Note that many of these benchmarks were developed to test a specific feature under a specific setting. We strive to keep the benchmarks up-to-date, but alas time is not an endless resource so some benchmarks will be outdated and irrelevant.

Running the benchmarks

As with testing, ts is used to run the benchmarks. Before running any benchmarks you have to install the tests. To get a listing of all benchmarks you have available call ts:benchmarks().

To run all benchmarks call ts:bench(). This will run all benchmarks using the emulator which is in your $PATH (Note that this does not have to be the same as from which the benchmarks were built from). All the results of the benchmarks are put in a folder in $TESTROOT/test_server/ called YYYY_MO_DDTHH_MI_SS.

Each benchmark is run multiple times and the data for all runs is collected in the files within the benchmark folder. All benchmarks are written so that higher values are better.

Writing benchmarks

Benchmarks are just normal testcases in Common Test suites. They are marked as benchmarks by being included in the AppName_bench.spec which is located in lib/AppName/test/ for the applications which have benchmarks. Note that you might want to add a skip clause to AppName.spec for the benchmarks if you do not want them to be run in the nightly tests.

Results of benchmarks are sent using the ct_event mechanism and automatically collected and formatted by ts.

ct_event:notify(
    #event{name = benchmark_data, 
            data = [{value,TPS}]}).

The application, suite and testcase associated with the value is automatically detected. If you want to supply your own you can include suite andor name with the data. i.e.

ct_event:notify(
    #event{name = benchmark_data, 
            data = [{suite,"erts_bench"},
                    {name,"ets_transactions_per_sec"},
                    {value,TPS}]}).

The reason for using the internal ct_event and not ct is because the benchmark code has to be backwards compatible with at least R14.

The value which is reported should be as raw as possible. i.e. you should not do any averaging of the value before reporting. The tools we use to collect the benchmark data over time will do averages, means, stddev and more with the data. So the more data which is sent using ct_event the better.

Viewing benchmarks

At the moment of writing this HOWTO the tool for viewing benchmark results is not available as opensource. This will hopefully change in the near future.