feature(BaseMonitorSet): Implement monitoring template

[k0machi]

This commit adds a new templating mechanism for monitoring dashboards,
allowing SCT to be in sync with changes in scylla-monitoring repo, by
first separating SCT metrics into a separate file, and then injecting
them into the overview template on the monitoring node, then
regenerating said template. This allows monitoring dashboard to always
be up to date with regards to changes on the monitoring side.

Fixes scylladb/qa-tasks#1444

Testing

• Jenkins

PR pre-checks (self review)

• I added the relevant backport labels
• I didn't leave commented-out/debugging code

Reminders

• Add New configuration option and document them (in sdcm/sct_config.py)
• Add unit tests to cover my changes (under unit-test/ folder)
• Update the Readme/doc folder relevant to this change (if needed)

[k0machi]

Demo:

One more thing that's missing are previously available template variables for event filtering - I'll add them soon.

[scylladbbot]

@k0machi new branch branch-2025.1 was added, please add backport label if needed

[k0machi]

Added annotations, template variables for event types, added logic to change default values (currently just hardcoded to change the by var, open for suggestions on the format if we want something more generic), re-arranged widgets to be more in line with the previous dash.

[k0machi]

This run has the updated dashboard, monitoring node is set to keep so will be alive for now.

[soyacz]

requests served per instance shows whole numbers (like 30000 instead of 30k) and miss unit (ops/s)

[k0machi]

requests served per instance shows whole numbers (like 30000 instead of 30k) and miss unit (ops/s)

Fixed

[k0machi]

I'll update the template based on recent changes by Valerii and then we can merge this.

[soyacz]

LGTM
Still docker provision test fails with c-s.
@dimakr do you know what could be the reason?

[fruch]

LGTM Still docker provision test fails with c-s. @dimakr do you know what could be the reason?

it's cause of counters, we need to drop them from those tests.

since we move to take latest scylla release (with tablets enabled by default), it's kind of broken
and forgot/didn't got to handle/report it

EDIT: I did handled a part of it, skipping the pre-create of counter table when tablets is enabled, but that wasn't enough

[k0machi]

Will rebase onto #10006 and update with changes from recent stress-command panels merge.

[k0machi]

Latest run that implements the new dash. Grafana node is still up.

[fruch]

LGTM