Add benchmark for the number of minimum cpu cores

[alexggh]

Fixes: #5122.

This PR extends the existing single core benchmark_cpu to also build a score of the entire processor by spawning EXPECTED_NUM_CORES(8) threads and averaging their throughput.

This is better than simply checking the number of cores, because also covers multi-tenant environments where the OS sees a high number of available CPUs, but because it has to share it with the rest of his neighbours its total throughput does not satisfy the minimum requirements.

TODO

• Obtain reference values on the reference hardware.

[paritytech-cicd-pr]

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable 2/3
Logs: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6802189

[sandreim]

A PRDoc is needed for operators. Otherwise LGTM!

[ggwpez]

I ran this 5 times on ref hardware and it is very consistent on 1022 MiBs for the new BLAKE2 parallel metric. Kind of expected on a homogeneous system 😆

[ggwpez]

PS: Apparently we did not update these metrics after devops moved the servers from GCP to Scaleway, so these numbers should still be fine for GCP servers:

If we update them now, then we basically raise the requirement from GCP default to something better. Personally i think we can keep it for now since node operators rely on this and use it quite often.

[alexggh]

PS: Apparently we did not update these metrics after devops moved the servers from GCP to Scaleway, so these numbers should still be fine for GCP servers:

You mean our current ref hardware it is not the same as the ones we generated reference_hardware.json from ?

If we update them now, then we basically raise the requirement from GCP default to something better. Personally i think we can keep it for now since node operators rely on this and use it quite often.

Yeah, I concur with you that we probably don't want to change the values for existing benchmarks, although I'm not sure where should I generate the new one I want to add.

[ggwpez]

You mean our current ref hardware it is not the same as the ones we generated reference_hardware.json from ?

Yea. We migrated the server in the meantime. But the definition of the ref HW in the wiki is still the same, so i think we should keep the values.

Yeah, I concur with you that we probably don't want to change the values for existing benchmarks, although I'm not sure where should I generate the new one I want to add.

Given that the multi core score was pretty much identical to the single thread score on the ref hardware, i think its fine to use the same value in the JSON config file. Any concerns about that?

[alexggh]

Yeah, I concur with you that we probably don't want to change the values for existing benchmarks, although I'm not sure where should I generate the new one I want to add.

Given that the multi core score was pretty much identical to the single thread score on the ref hardware, i think its fine to use the same value in the JSON config file. Any concerns about that?

Yeah, that should be fine as well, although I will try to see if I can get my hands on this type of machine.

[ggwpez]

You can request access by opening an issue here: https://github.com/paritytech/devops/issues. This is the machien: https://github.com/paritytech/devops/pull/3210

[alexggh]

PS: Apparently we did not update these metrics after devops moved the servers from GCP to Scaleway, so these numbers should still be fine for GCP servers:

If we update them now, then we basically raise the requirement from GCP default to something better. Personally i think we can keep it for now since node operators rely on this and use it quite often.

Did some measurements and spelunking on the following hardware-types and what I can tell is that the speed-up in benchmark comes actually from the fact that we introduced SIMD optimisations on the benchmarked functions here
Blake2 and SR25519-Verify without updating the reference performance.

• GCP: n2-standard: recommended here: https://wiki.polkadot.network/docs/maintain-guides-how-to-validate-polkadot#reference-hardware and it seems to be the machine type we previously used for benchmarking.

Master

+----------+-----------------------+-------------+-------------+-------------------+
| Category | Function              | Score       | Minimum     | Result            |
+==================================================================================+
| CPU      | BLAKE2-256            | 1.00 GiBs   | 783.27 MiBs | ✅ Pass (131.3 %) |
|----------+-----------------------+-------------+-------------+-------------------|
| CPU      | BLAKE2-256-Parallel-8 | 1.00 GiBs   | 783.27 MiBs | ✅ Pass (131.1 %) |
|----------+-----------------------+-------------+-------------+-------------------|
| CPU      | SR25519-Verify        | 701.33 KiBs | 560.67 KiBs | ✅ Pass (125.1 %) |

Without SIMD optimisations

+----------+-----------------------+-------------+-------------+-------------------+
| Category | Function              | Score       | Minimum     | Result            |
+==================================================================================+
| CPU      | BLAKE2-256            | 805.98 MiBs | 783.27 MiBs | ✅ Pass (102.9 %) |

• Scaleway machine type that we currently use for reference hardware, from here https://github.com/paritytech/devops/pull/3210

Master

+==================================================================================+
| CPU      | BLAKE2-256            | 969.47 MiBs | 783.27 MiBs | ✅ Pass (123.8 %) |
|----------+-----------------------+-------------+-------------+-------------------|
| CPU      | BLAKE2-256-Parallel-8 | 966.19 MiBs | 783.27 MiBs | ✅ Pass (123.4 %) |
|----------+-----------------------+-------------+-------------+-------------------|
| CPU      | SR25519-Verify        | 583.56 KiBs | 560.67 KiBs | ✅ Pass (104.1 %) |
|----------+-----------------------+-------------+-------------+-------------------|

Without SIMD optimizations

+----------+-----------------------+-------------+-------------+-------------------+
| Category | Function              | Score       | Minimum     | Result            |
+==================================================================================+
| CPU      | BLAKE2-256            | 782.02 MiBs | 783.27 MiBs | ✅ Pass ( 99.8 %) |

• AWS: c6i-4xlarge: recommended here: https://wiki.polkadot.network/docs/maintain-guides-how-to-validate-polkadot#reference-hardware
  ** Master **

+==================================================================================+
| CPU      | BLAKE2-256            | 1.03 GiBs   | 783.27 MiBs | ✅ Pass (134.3 %) |
|----------+-----------------------+-------------+-------------+-------------------|
| CPU      | BLAKE2-256-Parallel-8 | 1.02 GiBs   | 783.27 MiBs | ✅ Pass (133.8 %) |
|----------+-----------------------+-------------+-------------+-------------------|
| CPU      | SR25519-Verify        | 655.97 KiBs | 560.67 KiBs | ✅ Pass (117.0 %) |
|----------+-----------------------+-------------+-------------+-------------------|

Consequences

Every weight update we did after Blake2 (March 2023) got merged gets the 20-30% cpu speed up, but every validator using the benchmarks to determine if their validator is in parameters gets a false OK, because the reference value have not been increased. That means that there is potential that the weights are understimated with around 20-30%.

What next ?

From my perspective we have a few options here:

1. We increase the reference hardware benchmarks to reflect the optimisations, the unfortunate immediate consequence of that is that every validator that is around baseline will fail the check, and from https://telemetry.polkadot.io/ these numbers don't seem to be negligible and people will probably not be happy to get this out of the sudden.
   Kusama

Polkadot

2. Regenerate the weights that got the speed up and future weights on a slower hardware closee to the baseline.
3. Do nothing now, since this might not be a problem(it's been like that for 1.5y) yet and just use the values with the speed up for the newly introduced parallel benchmark BLAKE2-256-Parallel-8 which isn't planned to be enforced right away. This practically enforce validators to slowly converge to hardware where the single core BLAKE2-256 is also in sync with the reference hardware we use for generating the weights.

@ggwpez @koute @PierreBesson, thoughts ?

[ggwpez]

Then i think we should bump the numbers. Otherwise we silently and accidentally reduced the single core requirements by merging these dependency updates and not updating them.

Good find, thanks for investigating!

[alvicsam]

For transparency, CI is still using GCP machines and we are not planning to change it at least until we finish the ci migration.

[alexggh]

For transparency, CI is still using GCP machines and we are not planning to change it at least until we finish the ci migration.

@alvicsam These #5196 updated numbers are pretty similar between GCP and scaleway, overall the conclusion is that the speedup did not came from changing cloud providers, but simply from the optimisations that the code suffered since the reference numbers were last updated.

[ggwpez]

Going to approve now so feel free to merge.

[alexggh]

Going to approve now so feel free to merge.

Thanks, my plan is to merge this only after https://polkadot.subsquare.io/referenda/1051 passes and I do the necessary updates in: https://wiki.polkadot.network/docs/maintain-guides-how-to-validate-polkadot#reference-hardware.

[alexggh]

Referenda passed: https://polkadot.subsquare.io/referenda/1051, wiki page updated with w3f/polkadot-wiki#6202, merging it ...