Feature Request: Throttling

[slava-vishnyakov]

Some kind of throttling would be nice, like

"Don't send to this output more than once every 5 minutes"

Because sometimes you have a lot of errors at the same time and Slack is overloaded.

Thanks!

[slava-vishnyakov]

As a quick hack for 1 minute throttling

  def register
    ....
    @last_message = Time.now - 60
  end # def register

  public
  def receive(event)
    return unless output?(event)
    return if Time.now - @last_message < 60

    payload_json = Hash.new
    payload_json['text'] = event.sprintf(@format)
    @last_message = Time.now

[madslyng]

I read your OP-post and we have the same concerns, but I'm going to write our whole situation just to be clear (so I will repeat some issues):

We are testing out this plugin now, on a DEV environment, but have concerns about the throttling once what we are working on goes into production.

Specifically, we are using this plugin to notify us on Slack if something bad occurs (Error, Warning, Fatal) - and sadly this never comes alone.

Slack writes on their website about Rate Limits: https://api.slack.com/docs/rate-limits

They seem to allow 1 message per second, and shorter bursts that exceed it.

I'm not sure how to handle this, but like you have suggested, throttling is a way to start, but wouldn't that just be a bottleneck in trouble-some scenarios ?

How about message-bundling ?
How about duplicate detection ?
How about a summary ?

[slava-vishnyakov]

Sorry, we have since moved on to a custom solution. Anyway throttling is optional, I mean the developer would have to understand that he's about to lose some messages that way.

For us it works pretty well - during the spikes when we can expect hundreds of messages per second - we get only 1 per 30 seconds. Though we have something like a "key" or "namespace", so you get only 1 per 30 sec in this namespace and 1 per 30 sec in other, etc.. So for example if you have a burst of low-priority messages - your high priority message will still get through.

We also use Redis for that, because we have multi-deployment (more than 10 servers), so that they can coordinate.

We are kind of happy with our solution. :)

[cyli]

@slava-vishnyakov Apologies, for missing this issue! It seems I hadn't been watching my own repo for some reason.

I was wondering if https://www.elastic.co/guide/en/logstash/current/plugins-filters-throttle.html would work for for throttling messages?

@sonicjolt I'm not entirely sure what you mean by message-bundling, but would something like https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html work for that?

These seem to be extremely useful filter plugin suggestions that would be useful for any kind of notification outpuut.

[madslyng]

@cyli I'll have a look at it and get back to you.

Btw, by message bundling, I mean, in one post to slack, you could bundle many messages into one. It's not not pretty, but it's an options.

[cyli]

@sonicjolt Ah ok, thanks. That might work - I hadn't tried it previously, but maybe something with message attachments?

It looks like there seems to be a general desire for this kind rate-limiting/bundling/deduping: https://www.bountysource.com/issues/3615044-handling-pagerduty-rate-limits.

The reason I suggest a filter plugin is that I'd imagine that probably a developer would want to customize the summary/bundling format, and possibly even compose rate-limiting, bundling, and summarizing all together.

In our deployment, we aggregate all logs to a redis queue, and we have a logstash process running that ships all of those messages to elasticsearch and pops events off the redis queue using data_type => list (which uses BLPOP), and a different logstash process that just listens to events in the redis queue using data_type => channel (which uses SUBSCRIBE) and pushes messages to slack. So each one can use different filter plugins without affecting each other.

I'm not sure how common this is though, and maybe having filter plugins is a non-starter because it would mutate what gets put into elasticsearch for most people?

Update: This is not a refusal to add any of this functionality to this plugin - believe the IRC plugin for instance does provide some rate limiting, because the IRC bot library that is used does. I'm just wondering if there is a more general solution or pattern.

[garlandkr]

Throttling is already implemented in other plugins as mentioned. This should be closed.

As mentioned by @cyli we also use filtering on the logstash-agents to decide when to send messages. The way the filter works in our case is that throttle adds a tag:

  if [level] == "ERROR" {
    throttle {
        before_count => -1
        after_count => 1
        period => 3600
        key => "%{service}%{host}"
        add_tag => "throttled"
    }
  }

Then in the output section it will send everything to Redis, and it will also send a notification if the throttled tag exists:

  redis{
    host=>[{% for instance in redis_instances.instances %}"{{ instance.private_ip }}"{% if not loop.last %},{% endif %}{% endfor %}]
    port=>6379
    db=>5
    data_type=>"list"
    key=>"logstash"
  }
  if [level] == "ERROR" and "throttled" not in [tags] {
    slack { 
       url => "{{ slack_channel_web_hook }}"
       format => "[%{service}][%{host}][{{ aws_environment|upper }}] %{message}"
    }
  }