Remove status metrics for deleted resources

[ryanrolds]

Description

The status metrics for resources are left around after the resource is deleted. If the resource was invalid, the metric would continue to report invalid causing false alarms that required restarting the controller to clear.

Sleeping for 1 second was required to avoid a race condition that resulted in the new metrics getting caught up in the clearing/unregistering. I don't like adding the sleep, but that only other option is upgrading the metrics package we are using to a newer supported one (which doesn't have the race condition, the other package use a mutex).

When we upgrade to a new package we can remove this. The E2E test should have long-term value.

Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works

[solo-changelog-bot]

Issues linked to changelog:
kgateway-dev#6938

[ryanrolds]

/kick-ci

[ryanrolds]

/kick-ci

[github-actions]

Visit the preview URL for this PR (updated for commit d69d642):

https://gloo-edge--pr10597-rolds-resource-metri-eq3r3xf3.web.app

_{(expires Tue, 18 Feb 2025 21:28:22 GMT)}

_{🔥 via Firebase Hosting GitHub Action 🌎}

_{Sign: 77c2b86e287749579b7ff9cadb81e099042ef677}

[ryanrolds]

/kick-ci

[ryanrolds]

/kick

[nfuden]

I dont quite understand the rationale for clearing all statuses rather than doing something a bit more targeted and am concerned it could cause some flicker in metrics.

If an extra example can be added to prove otherwise then I am onboard

[nfuden]

cant this cause flickering to an operator?
Why cant we set new statuses and then clear all out of compliance ones after?

[ryanrolds]

We are not able to clear out metrics at the label/attribute level. Only at the metric-level (view), which includes all labels).

[ryanrolds]

I dont quite understand the rationale for clearing all statuses rather than doing something a bit more targeted and am concerned it could cause some flicker in metrics.

The API provided by the metrics packages just don't provide the level of access needed to filter/remove specific label combinations from the metrics/views. I wish it wasn't that way. The same issue exists in the package we would like to move to, but it won't require a sleep reducing the data race we are both concerned about.