Add alert when there are too many queued jobs on crates.io #330
Conversation
|
The existing alert is "a job was queued for more than an hour", which needlessly delayed on-call getting paged. |
| @@ -188,6 +188,12 @@ | |||
| labels: | |||
| dispatch: pagerduty-cratesio | |||
|
|
|||
| - alert: TooManyQueuedJobs | |||
| expr: cratesio_service_background_jobs{env="prod"} > 100 | |||
There was a problem hiding this comment.
we introduced a priority column for background jobs lately and when we backfilled the rust-version values a few weeks ago we had several thousand low-priority background jobs queued up. I guess such a case would trigger the pagerduty alert if we merged it like this. Maybe we should somehow feed the priority into the monitoring system too? For the bin/monitor application we only consider priority >= 0 (normal or higher).
There was a problem hiding this comment.
I don't think the priority is exposed through the cratesio_service_background_jobs metric though. We should add it first.
There was a problem hiding this comment.
Will update once rust-lang/crates.io#6871 is deployed.
There was a problem hiding this comment.
the PR is deployed in staging and production now
cc @rust-lang/crates-io