Benchmark performance against brian2genn and cpp_standalone

[denisalevi]

This should be the first benchmarking (speed tests) against brian2genn and cpp_standalone after updating brian2cuda to use brian2 2.0 (#22), but before major optimizations of brian2genn.
It should include automated benchmarking scripts to compare effects of later optimizations.

[denisalevi]

Our benchmark folder has now speed test results from before no_or_const_delay_mode implementation, after and with additional profiling_info collected using cudaEvents.

I have until now only benchmarked against CPPStandaloneConfiguration and GeNNConifguration. But there are also CPPStandaloneConfigurationOpenMP and GeNNConfigurationOptimized.

@moritzaugustin Do we want to benchmark agains OpenMP as well?

@mstimberg Did you experience any "optimization" in the GeNNConfigurationOptimized? I did run that when I first ran the speed tests but could not see any performance difference to GeNNConfiguration and the brian2genn benchmarkfolder does not seem to have used it either. So can we just us GeNNConfiguration?

[mstimberg]

@mstimberg Did you experience any "optimization" in the GeNNConfigurationOptimized? I did run that when I first ran the speed tests but could not see any performance difference to GeNNConfiguration and the brian2genn benchmarkfolder does not seem to have used it either. So can we just us GeNNConfiguration?

This is a bit misleading, I agree. The only difference between the two configurations is that GeNNConfiguration switches off the default compiler args that we normally pass to gcc (-w -O3 -ffast-math -fno-finite-math-only -march=native). This only affects the host code that is run on the CPU and I don't think any of the benchmarks spends any significant time in this code, so I don't expect it to have a measurable effect on performance. However, I'd probably just use GeNNConfigurationOptimized instead of GeNNConfiguration, simply because the "optimized" configuration uses the defaults (i.e. what a user would get if they don't change any preferences).

[denisalevi]

We have benchmarking scripts implemented for a while now. Closing this issue.