Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to exlcude alerts from the results #67

Merged
merged 1 commit into from
Dec 19, 2024
Merged

Conversation

martialblog
Copy link
Member

This adds an option to exclude alerts from the results.

The use case behind this: sometimes you want to define
a so called Watchdog or DeadMansSwitch alert that is always
firing, in order to monitoring that the alerting is working.

When such a Watchdog is defined the list of all alerts will
always be Critical. Thus we add a flag to exclude certain alerts.

See #61

I want to tackle the issue in two steps, this is the first. The second is a feature to specify the expected exit code.

@martialblog
Copy link
Member Author

How often can a person misspell "exclude"! 🤦

@martialblog
Copy link
Member Author

@RincewindsHat The commit is ontop the others since otherwise I would have to cleanup the merge conflict and I'm so so so so lazy. Just ignore the others

@martialblog martialblog added this to the v0.3.0 milestone Dec 16, 2024
@wattebausch
Copy link

Looks very good. Even works with multiple excludes

Check with "alert -P (--problems)"

./check_prometheus -H metrics.some-probelm-cluster.example.com -p 80 alert -P 
[CRITICAL] - 1 Alerts: 1 Firing - 0 Pending - 0 Inactive
\_ [CRITICAL] [Watchdog] is firing - value: 1.00

 ./check_prometheus -H metrics.some-probelm-cluster.example.com -p 80 alert -P --exclude-alert Watchdog
[UNKNOWN] - 0 Alerts: 0 Firing - 0 Pending - 0 Inactive

Only "alert"

./check_prometheus -H metrics.some-probelm-cluster.example.com -p 80 alert 
[CRITICAL] - 126 Alerts: 1 Firing - 0 Pending - 125 Inactive
\_ [OK] [FluxHelmReleaseNotReady] is inactive
..
\_ [OK] [TargetDown] is inactive
\_ [CRITICAL] [Watchdog] is firing - value: 1.00
\_ [OK] [InfoInhibitor] is inactive


./check_prometheus -H metrics.slfl28652.wsl.ch -p 80 alert --exclude-alert Watchdog
[OK] - 125 Alerts: 0 Firing - 0 Pending - 125 Inactive
\_ [OK] [FluxHelmReleaseNotReady] is inactive
..
\_ [OK] [TargetDown] is inactive
\_ [OK] [InfoInhibitor] is inactive
..

no impact if Cluster is allready okay. verry nice

 ./check_prometheus -H metrics.clean-cluster.example.com -p 80 alert -P --exclude-alert Watchdog
[UNKNOWN] - 0 Alerts: 0 Firing - 0 Pending - 0 Inactive

/check_prometheus -H metrics.clean-cluster.example.com -p 80 alert --exclude-alert Watchdog
[OK] - 125 Alerts: 0 Firing - 0 Pending - 125 Inactive
\_ [OK] [FluxHelmReleaseNotReady] is inactive
..
\_ [OK] [TargetDown] is inactive
\_ [OK] [InfoInhibitor] is inactive

cmd/alert.go Outdated
// Matches a list of regular expressions against a string.
func matches(input string, regexToExclude []string) bool {
for _, regex := range regexToExclude {
re := regexp.MustCompile(regex)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not ideal. If the regex is invalid (as hard as it might be to achieve that), this line panics.
I think I would prefere a controled exit with a helpful message here.
the ..MustCompile functions are preferable for static expressions.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, you're right. I updated the implementation

@martialblog martialblog marked this pull request as ready for review December 18, 2024 17:36
The use case behind this: sometimes you want to define
a so called Watchdog or DeadMansSwitch alert that is always
firing, in order to monitoring that the alerting is working.

When such a Watchdog is defined the list of all alerts will
always be Critical. Thus we add a flag to exclude certain alerts.
Copy link
Member

@RincewindsHat RincewindsHat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good :-)

@martialblog martialblog merged commit 6d8740a into main Dec 19, 2024
2 checks passed
@martialblog martialblog deleted the exclude-list branch December 19, 2024 08:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants