Mention 2~3 ns benchmarks

[asterite]

Hi!

Really nice project. It helps to decide what to use.

Just a note: when benchmark iteration times are around 2~3 nanoseconds then there's probably really no difference at all. You can try this by swapping the order of the benchmark blocks, you always get the last one a bit slower than the first one. I'm not sure why this happens, though. But it would be nice to clarify that "in this benchmark it makes no difference what you use because (the reason I'm explaining here)".

This is just to make sure people don't use array[0] instead of array.first because they think it's faster. It's not.

Thank you!

[konung]

Thank you. Good point. I'll look into it. And maybe in some kind of automation as well

[fernandes]

hey @asterite

just a question regarding your point, in some cases like this one that compares String#dup vs String#+

here is the result:

 String#+@ 471.96M (  2.12ns) (± 9.13%)  0 B/op   1.07× slower
String#dup 502.68M (  1.99ns) (± 5.32%)  0 B/op        fastest

the first one, is a little bit slower, so should we consider String#dup is faster than String#+ because they are pretty close, but the slower is the first, not the second one?

@konung maybe in these ns cases, should increse the order of executions, to filter some "noise" when running the benchmark?

make sense both points?

[asterite]

@fernandes What happens if you invert the order of the benchmark? First dup, then +?

[fernandes]

 String#+@ 537.45M (  1.86ns) (± 7.64%)  0 B/op   1.05× slower
String#dup 566.09M (  1.77ns) (± 3.70%)  0 B/op        fastest

String#dup  548.3M (  1.82ns) (± 7.15%)  0 B/op   1.03× slower
 String#+@ 563.94M (  1.77ns) (± 3.99%)  0 B/op        fastest

uow, now the first is the slower hahaha

[fernandes]

ips has a default of: calculation = 5, warmup = 2, now I'm doing calculation = 20, warmup = 2, it's gonna take a little bit longer, but just for this test, it's ok

my idea is try to avoid any stdev running, specially because any process (even Chrome) that preempts the processor in 5s duration can influence the outcome, no?

Now with 20/2

String#dup 526.74M (   1.9ns) (± 9.23%)  0 B/op   1.07× slower
 String#+@ 564.07M (  1.77ns) (± 4.86%)  0 B/op        fastest

 String#+@ 502.29M (  1.99ns) (±10.27%)  0 B/op   1.11× slower
String#dup 557.57M (  1.79ns) (± 4.70%)  0 B/op        fastest

maybe the benchmarks should run each time changing the first execution, if the result invert, no conclusion can be made