Conditional Threshold rule and math aggregations in Rule file

[firdausa-iprogrammer]

Hi Team,

Is there a way to -
Set conditional values in Thresholds?
For eg. L1 alerts should only be triggered for a count of 5 and the alerts should only go to the L1 team.
L2 alerts should trigger at a count of 10 and again alerts should go only to the respective team.

Run filter ratio or math aggregations in the rule file itself to calculate percentage from an integer/double/float field.
Eg. Success count of hits from an API divided by the total count of hits from an API multiplied by 100 to get success %

[jertel]

I'm reading two questions in this post:

1. How do I send different alerts depending on the threshold limits?
2. How do I calculate a success percentage?

For #1, I suggest using two rules, with rule A having a threshold of 5, and rule B having a threshold of 10. Rule A alerts go to email group X, and rule B alerts go to email group Y. When the count exceeds threshold of 5, only group X receives the alert. When the count exceeds threshold of 10, both groups X and Y receive the alert. As far as I know it is not possible to stop the alerts going to the first group X once the count continues exceeding the lower threshold.

For #2, see the percentage_match example for a sample rule that attempts what you're requesting.

[firdausa-iprogrammer]

Hi @jertel,

You're spot on about the questions.
For 1. As you pointed out there is the limitation that both groups X and Y receive the alerts also the problem I'm looking to avoid is excess CPU and RAM utilization, if I'm not mistaken every rule configured runs it's own Elasticsearch query correct?
And the issue is that my team wants to setup something like 91 rules(currently) with 6 management levels.
In that case we're going to have 91 * 6 rules for a total of 546 rules which means 546 rules firing queries at Elasticsearch at one time. Hence the request for the conditionals.

For 2. What I need to do is get a far more advanced version of the percentage_match rule.
In TSVB's Filter ratio what I do is first get the Total count(Success + failure) then use a filter to remove either successes or failures to get the success or failure percentage.
In Timelion I've actually configured ES queries to achieve the same.

.es(index=logstash-*,metric=count,q='NOT document.APIResponse.data.errorCode: API_SUCCESS_RESP_TIME-007', split=document.APIResponse.data.errorMessage.keyword:5).divide(.es(index=logstash-*,metric=count,q='document.APIResponse.data.errorCode: *')).multiply(100).fit(mode=nearest).lines(fill=2,width=1)

In this case the errorCode of API_SUCCESS_RESP_TIME-007 is our successes, it's then divided by the failures and multiplied by 100.

From what I've seen percentage_match only manages simple percentage aggregations, do correct me if I'm wrong

[ferozsalam]

I think @jertel is correct about the percentage_match being what you want.

You need to define the query for successes and failures in the filter field.

You then need to define the query for just the successes in the match_bucket_filter field.

Assuming the min_percentage is 95%, the alert would then fire if less that 95% of responses were successful. Does this make sense? Something like:

filter:
- query:
    query_string:
      query: "document.APIResponse.data.errorCode:*"

match_bucket_filter:
- query:
    query_string:
      query: "document.APIResponse.data.errorCode: API_SUCCESS_RESP_TIME-007"

N.B. I haven't actually tested the above so it might need some tweaking to get working.

[firdausa-iprogrammer]

This might work, I'll give it a shot, thanks a ton again!!

[ferozsalam]

Hey @firdausa-iprogrammer - if as you suggest in #312 (reply in thread) you have a large number of management levels and don't want to duplicate the conditional logic, I think your best option at the moment is to write a custom alerter.

This is quite easy - see here for documentation and here for an example of a simple alerter - and you can easily inherit from a parent alerter class. So what I'm proposing is that - for example - if you were looking to alert into Slack, you could define the rule you need exactly as you would define any other Slack rule, and have a custom alerter that is a subclass of the Slack alerter.

You can then check the length of the matches array to see how many ES documents your query matched, and populate the Slack webhook URL in self.rule based on what management level should be contacted given the number of matches. Then just call the parent Slack alerter, which will handle the logic of actually firing into Slack.

In terms of the broader ElastAlert project, any threshold-based behaviour would have to be carefully thought out and implemented in as generic a manner as possible to avoid confusing behaviour across alerters. I doubt this will be implemented in the near future, so I think going the custom route is the quickest and easiest way to what you need.

We use a custom alerter in my organisation to achieve a simple level of priority-based behaviour across different alerting platforms, which is similar to what you suggest. I'm happy to help answer questions about implementation if you go down this route.

[firdausa-iprogrammer]

@ferozsalam That's brilliant, thanks!
Between a custom alert and a custom rule which do you think would be easier to implement?
https://elastalert2.readthedocs.io/en/latest/recipes/adding_rules.html

And any ideas for the % based point?

[ferozsalam]

I don't think there's much difference between the two in terms of implementation difficulty, but I think that logically what you're looking for is a different type of Alerter behaviour rather than a different type of Rule behaviour, so to me it makes more sense to extend an Alerter.

[firdausa-iprogrammer]

Hi @jertel and @ferozsalam,

Is it possible to extend the command alerter?
I use signal-cli to send alerts to a signal group.

How would my logic change in that case?

[ferozsalam]

This is probably a question for an entirely different discussion topic.

That said, what have you tried so far? I assume you have read the documentation here. If you show us what your current state is - what your rule looks like and any error messages you are receiving, we will be better able to help you.

[firdausa-iprogrammer]

To answer myself -

Here's my custom alerter -

from elastalert.alerts import Alerter
import subprocess
from elastalert.util import resolve_string

class fidAlerter(Alerter):
  required_options = set(['command_l1', 'command_l2', 'command_l3', 'thresh_l1_max', 'thresh_l2', 'thresh_l2_max'])
  def alert(self, matches):
      for match in matches:
        pct = match['percentage']
        thresh_l1 = self.rule['max_percentage']
        thresh_l1_max = self.rule['thresh_l1_max']
        thresh_l2 = self.rule['thresh_l2']
        thresh_l2_max = self.rule['thresh_l2_max']
        if pct >= thresh_l1 and pct <= thresh_l1_max:
          self.rule['command_l1'] = [self.rule['command_l1']]
          signal = [resolve_string(command_l1, matches[0]) for command_l1 in self.rule['command_l1']]
          subprocess.run(signal, shell=True)
        elif pct >= thresh_l2 and pct <= thresh_l2_max:
          self.rule['command_l2'] = [self.rule['command_l2']]
          signal = [resolve_string(command_l2, matches[0]) for command_l2 in self.rule['command_l2']]
          subprocess.run(signal, shell=True)
        else:
          self.rule['command_l3'] = [self.rule['command_l3']]
          signal = [resolve_string(command_l3, matches[0]) for command_l3 in self.rule['command_l3']]
          subprocess.run(signal, shell=True)

  def get_info(self):
    return {'type': 'Threshold Alerter'}

RULE CHANGES

max_percentage: 1
thresh_l1_max: 5
thresh_l2: 5
thresh_l2_max: 10

command_l1: 
command_l2:
command_l3:

@ferozsalam and @jertel If you could give me any suggestions to improve this it would be much appreciated!
Please mark whichever answer you feel like.
Thanks a ton for both of your constant help and support!

[jertel]

Review your logic for when pct == 5. Because of the use of > and < instead of >= or <= you will end up having a 5% match fall into L3.

[firdausa-iprogrammer]

Right, yes that makes sense, I'll change it to >= and <= instead.

Edited my script as well

Thanks!!

[firdausa-iprogrammer]

@jertel
Please feel free to mark this as the answer, I don't seem able to.

[firdausa-iprogrammer]

Hi!

So sorry to bug you guys about this again, my company now wants to switch from Signal to Telegram.
I'm still able to use the alerter in the answer above, I have a minor error however -

When I was using signal-cli for alerts \n worked to get the text in a new line.
For Telegram I'm using curl instead of the alerter that's already present to achieve my goal of threshold based alerting.
My curl command -

curl -XPOST https://api.telegram.org/bot<botToken>/sendMessage?chat_id=<chat_id> -d "text=- Alert_level: L1(Devops)                                                                                                         
   - Alert_Type: Every 10 minutes 
  - APIName: <alert name>
  - etc

When I fire this same command from bash even with sh -c "" which I believe subprocess uses I get the output on Telegram perfectly fomatted I don't even need \n.

However when I trigger the alert from elastalert it sends all the data but with only about 2 character space
for eg.

- Alert_level: L1(Devops)  - Alert_Type: Every 10 minutes - APIName: <APIName>

How do I format my curl command to work like it works on it's own?

[jertel]

Hello! I suggest including the snippet from your config showing your command alerter parameters so that someone here can better assist.

[firdausa-iprogrammer]

Hi @jertel,

The alerter script is in this discussion itself.
And I've attached the curl command as well.

[firdausa-iprogrammer]

command_l1: curl -XPOST https://api.telegram.org/bot<botToken>/sendMessage?chat_id=<chat_id> -d 'text="- Alert_level: L1-Devops \n
  - Alert_Type: Every 10 minutes \n
  - APIName: <APIName> \n
  - ErrorCode: <ErrorCode> \n
  - Alert start time: {starttime_ist} \n
  - Alert end time:   {endtime_ist} \n
  - Threshold: 5 %% in the Last 10 minutes \n
  - Current_Failure_%% in the Last 10 minutes: {percentage:.2f} %% \n
  - Error_Message: <Error_Message> \n
  - ErrorDescription: <ErrorDescription> \n
  - SOR: <SOR> \n
  - Action to be taken: \n
    1. Automated Notification Email sent to L1-Devops. \n
    2. SOR team must revert in next 2 hrs \n
    3. if revert not received then DevOps team contact SOR team for update \n
    4. Identify the issue & update on same in signal grp"' 

[jertel]

This should be a new discussion, with an optional link back to the original discussion.

See #11 which requests that old discussions not be reused.

[firdausa-iprogrammer]

Apologies!
I'll set this up as a new discussion.