Some alerts stuck in firing state

[paxlo]

What went wrong?

What happened:

• Some random alerts are sometimes stuck in firing state, although in alermanager their status is resolved.

What did you expect to happen:

• If an alert has been resolved in alertmanager, it sould have the same status in oncall

How do we reproduce it?

Can't reproduce

Grafana OnCall Version

1.11.3

Product Area

Alert Flow & Configuration

Grafana OnCall Platform?

Kubernetes

User's Browser?

No response

Anything else to add?

I suspect it may have something to do with redis.

[Walkablenormal]

There is not a lot to go on in this issue but im facing similar problems. It mostly seems to happen when alerts in Grafana IRM are firing for a long time, and then get resolved. Even when Grafana OnCall is doing a sync, it does not remove the alert from OnCall. There is nothing in the logs that seem to indicate any authentication or connection error.

Do I understand it correctly that the periodic sync should detect differences between Grafana IRM and OnCall and resolve/remove alerts that are still firing in OnCall when they are resolved/missing from IRM?

Let me know if I need to provide logging to debug this issue.