In same rule have multiple messages alert, but only one of them is firing in Alertmanager.

[thangtq710]

Hi, in my Elastalert rule:
test.yaml:

name: test3
type: percentage_match
index: apm-transaction-*
buffer_time:
  minutes: 10
run_every:
  minutes: 1
filter:
- term:
    service.environment: "DEV"
query_key: service.name
match_bucket_filter:
- term:
    event.outcome: "failure"
max_percentage: 2
realert:
  minutes: 0
alert:
  - "alertmanager"
alert_text_type: alert_text_only
alert_text: "Test3 esalert with service {0} percentage {1}"
alert_text_args: ["service.name", "percentage"]
alertmanager_hosts:
  - "http://alertmanager:9093"
alertmanager_api_version: "v2"
alertmanager_annotations:
  severity: "critical"
alertmanager_labels:
  project: "elastalert"
  alertname: "test3"

Ouput command elastalert-test-rule rules/test.yaml

INFO:elastalert:Alert for test at 2024-08-05T09:24:26.680286Z:
INFO:elastalert:Test3 esalert with service dev-service-helper percentage 100.0

INFO:elastalert:Alert for test at 2024-08-05T09:44:26.680286Z:
INFO:elastalert:Test3 esalert with service dev-service-a percentage 7.69

INFO:elastalert:Alert for test at 2024-08-05T09:44:26.680286Z:
INFO:elastalert:Test3 esalert with service dev-service-b percentage 8.33

File config alertmanager.yaml:

global: {}
receivers:
- name: default-receiver
- name: test_alert
  webhook_configs:
  - send_resolved: true
    url: http://test/elastalert
route:
  group_by:
  - test
  group_interval: 3m
  group_wait: 10s
  receiver: default-receiver
  repeat_interval: 3h
  routes:
  - continue: true
    group_by:
    - alertname
    - project
    group_interval: 2m
    group_wait: 10s
    matchers:
    - alertname="test3"
    - project="elastalert"
    receiver: test_alert
    repeat_interval: 10m

But Alertmanager only one of them is firing an alert:

I need firing all message:

Test3 esalert with service dev-service-helper percentage 100.0
Test3 esalert with service dev-service-a percentage 7.69
Test3 esalert with service dev-service-b percentage 8.33.

Please help me, thanks you !

[jertel]

I suggest enabling debug logging so you can observe what's happening internally. It will show the request and the response from Elasticsearch.

[thangtq710]

Thanks you, Because Alertmanager is grouping alerts with the same alertname and project labels together into a single alert. I add a unique label to each alert and resolve.

alertmanager_labels:
  project: "elastalert"
  alertname: "test3"
alertmanager_fields:
  service: service.name