feat: new template counting and phase counting metrics

[Joibel]

From #12589.

New metric total_count which is like the old count metric and the new gauge metric, but a counter, not a gauge. The gauge shows a snapshot of what is happening right now in the cluster, the counter can answer questions like how many Failed workflows have there been in the last 24 hours.

Two further metrics for counting uses of WorkflowTemplates via workflowTemplateRef only. These store the name of the WorkflowTemplate or ClusterWorkflowTemplate if the cluster_scope label is true, and the namespace where it is used. workflowtemplate_triggered_total counts the number of uses. workflowtemplate_runtime records how long
each phase the workflow running the template spent in seconds.

Note to reviewers: this is now a standlone commit

[agilgur5]

Note to reviewers: this is part of a stack of reviews for metrics changes. Please don't merge until the rest of the stack is also ready.

@Joibel since these ones aren't draft, does that mean they're ready to review but not ready to merge?

What we can do to indicate that is to add a "[DO NOT MERGE] " prefix to the title of each of these -- since that will fail the PR title check, it won't be mergeable and is a pretty big indicator too. Or "[Please Review but DO NOT MERGE] " or something

[Joibel]

These are ready for review.

Merging isn't a disaster, it just makes any merged into PRs slightly harder to review. I was hoping anyone reviewing would do enough reading not to merge with the existing words I've written.

The whole thing is an experiment in highly stacked PRs from my point of view.

[agilgur5]

I was hoping anyone reviewing would do enough reading not to merge with the existing words I've written.

Unfortunately, I'm not sure that's always the case 😕 But more importantly, there's also the muscle memory of the "Approve" -> "Merge" process that makes it more likely to happen unintentionally. As such, I'm gonna mark them as "[DNM]" and maybe put an emoji indicating reviewed or not for at a glance filtering. Also "[1/12]" would work too as a blocking prefix

[isubasinghe]

Just a small comment