Test Rule question

[roman-tasi]

I am running my test rule command and am getting this:

elastalert_status - {'rule_name': 'Lockout Alerting', 'endtime': datetime.datetime(2021, 9, 28, 18, 52, 47, 137084, tzinfo=tzutc()), 'starttime': datetime.datetime(2021, 9, 28, 17, 52, 11, 137084, tzinfo=tzutc()), 'matches': 76742, 'hits': 76742, '@timestamp': datetime.datetime(2021, 9, 28, 18, 54, 48, 709983, tzinfo=tzutc()), 'time_taken': 118.85222268104553}

Basically I don't know why my matches is equal to my hits. Also it isn't logical for me to have 76742 matches. Also the sample rule command is printing this out repeatedly:

silence - {'exponent': 0, 'rule_name': 'Lockout Alerting.10.100.10.83', '@timestamp': datetime.datetime(2021, 9, 28, 18, 54, 46, 435723, tzinfo=tzutc()), 'until': datetime.datetime(2021, 9, 28, 18, 55, 46, 435708, tzinfo=tzutc())}

but obviously with different IPs and timestamps, but all have the silence tag.

Can anyone help?

Here is my rule config (ssh.yaml):

  name: Lockout Alerting

  type: frequency

  num_events: 1

  timeframe:
    minutes: 60

  - query:
      query_string:
        query: "A user account was locked out."

  index: winlogbeat-*

  query_key:
    - source.ip

  include:
    - host.hostname
    - user.name
    - source.ip

  include_match_in_root: true

  alert_subject: "SSH abuse on <{}> | <{}|Show Dashboard>"
  alert_subject_args:
    - host.hostname
    - kibana_link

  alert_text: |-
    An attack on {} is detected.
    The attacker looks like:
    User: {}
    IP: {}
  alert_text_args:
    - host.hostname
    - user.name
    - source.ip

  alert_subject: "Lockout Alert"

  alert:
  - "email"
  email:
  - "info@hotmail.com"

  use_kibana4_dashboard: "https://dev.somewhere/app/kibana#/dashboard/37739d80-a95c-11e9-b5ba-33a34ca252fb"

These are three sample logs that we our running our rule against:

`Oct 5, 2021 @ 10:30:26.059 | @timestamp:Oct 5, 2021 @ 10:30:26.059 @Version:1 agent.ephemeral_id:9dc9c78a-6233-4199-9647-dc67ee366567 agent.hostname:Dcon3 agent.id:15c7c4f2-ef0b-4828-9a2d-78d6f3d81d49 agent.name:Dcon3 agent.type:winlogbeat agent.version:7.10.0 ecs.version:1.5.0 event.action:logged-out event.category:authentication event.code:4634 event.created:Oct 5, 2021 @ 10:30:26.404 event.kind:event event.module:security event.outcome:success event.provider:Microsoft-Windows-Security-Auditing event.type:end host.name:Dcon3.uhtasi.local log.level:information message:An account was logged off. Subject: Security ID: S-1-5-21-3693063922-990322624-829951171-2161 Account Name: netmon Account Domain: UHTASI Logon ID: 0x4C7FCCB5 Logon Type: 3 This event is generated when a logon session is destroyed. It may be positively correlated with a logon event using the Logon ID value. Logon IDs are only unique between reboots on the same computer. related.user:netmon tags:beats_input_codec_plain_applied type:winlogbeat user.domain:UHTASI user.id:S-1-5-21-3693063922-990322624-829951171-2161 user.name:netmon user.name.text:netmon winlog.api:wineventlog winlog.channel:Security winlog.computer_name:Dcon3.uhtasi.local winlog.event_data.LogonType:3 winlog.event_data.TargetDomainName:UHTASI winlog.event_data.TargetLogonId:0x4c7fccb5 winlog.event_data.TargetUserName:netmon winlog.event_data.TargetUserSid:S-1-5-21-3693063922-990322624-829951171-2161 winlog.event_id:4634 winlog.keywords:Audit Success winlog.logon.id:0x4c7fccb5 winlog.logon.type:Network winlog.opcode:Info winlog.process.pid:704 winlog.process.thread.id:892 winlog.provider_guid:{54849625-5478-4994-A5BA-3E3B0328C30D} winlog.provider_name:Microsoft-Windows-Security-Auditing winlog.record_id:56772934 winlog.task:Logoff _id:DZ4mUnwBkDknLaJi3J0m _index:winlogbeat-7.10.0-2021.10.04-000127 _score: - _type:_doc

| Oct 5, 2021 @ 10:30:26.022 | @timestamp:Oct 5, 2021 @ 10:30:26.022 @Version:1 agent.ephemeral_id:dbb2fa17-f4a8-4959-8d84-0a8986c9aab1 agent.hostname:DCON2 agent.id:564b28a5-8795-4352-a09c-d5782975841b agent.name:DCON2 agent.type:winlogbeat agent.version:7.10.0 destination.domain:dcon3.uhtasi.local destination.ip:10.100.14.8 destination.port:59649 ecs.version:1.5.0 event.action:Network connection detected (rule: NetworkConnect) event.category:network event.code:3 event.created:Oct 5, 2021 @ 10:29:09.268 event.kind:event event.module:sysmon event.provider:Microsoft-Windows-Sysmon event.type:connection, start, protocol host.name:DCON2.uhtasi.local log.level:information message:Network connection detected: UtcTime: 2021-10-05 20:30:26.022 ProcessGuid: {26AFC6E6-D13F-612E-0000-0010BA3E0200} ProcessId: 1336 Image: C:\Windows\System32\dns.exe User: NT AUTHORITY\SYSTEM Protocol: udp Initiated: false SourceIsIpv6: false SourceIp: 10.100.14.8 SourceHostname: DCON2.uhtasi.local SourcePort: 53 SourcePortName: domain DestinationIsIpv6: false DestinationIp: 10.100.14.8 DestinationHostname: dcon3.uhtasi.local DestinationPort: 59649 DestinationPortName: network.community_id:1:kPgB6sGJLXOuH5R+RwD51733qEE= network.direction:inbound network.transport:udp network.type:ipv4 process.entity_id:{26AFC6E6-D13F-612E-0000-0010BA3E0200} process.executable:C:\Windows\System32\dns.exe process.executable.text:C:\Windows\System32\dns.exe process.name:dns.exe process.name.text:dns.exe process.pid:1336 related.ip:10.100.14.8, 10.100.14.8 related.user:SYSTEM source.domain:DCON2.uhtasi.local source.ip:10.100.14.8 source.port:53 tags:beats_input_codec_plain_applied type:winlogbeat user.domain:NT AUTHORITY user.name:SYSTEM user.name.text:SYSTEM winlog.api:wineventlog winlog.channel:Microsoft-Windows-Sysmon/Operational winlog.computer_name:DCON2.uhtasi.local winlog.event_data.SourcePortName:domain winlog.event_id:3 winlog.opcode:Info winlog.process.pid:2,756 winlog.process.thread.id:3,464 winlog.provider_guid:{5770385F-C22A-43E0-BF4C-06F5698FFBD9} winlog.provider_name:Microsoft-Windows-Sysmon winlog.record_id:546206126 winlog.task:Network connection detected (rule: NetworkConnect) winlog.user.domain:NT AUTHORITY winlog.user.identifier:S-1-5-18 winlog.user.name:SYSTEM winlog.user.type:User winlog.version:5 _id:S54lUnwBkDknLaJi6Y1a _index:winlogbeat-7.10.0-2021.10.04-000127 _score: - _type:_doc

| Oct 5, 2021 @ 10:30:25.774 | @timestamp:Oct 5, 2021 @ 10:30:25.774 @Version:1 agent.ephemeral_id:9dc9c78a-6233-4199-9647-dc67ee366567 agent.hostname:Dcon3 agent.id:15c7c4f2-ef0b-4828-9a2d-78d6f3d81d49 agent.name:Dcon3 agent.type:winlogbeat agent.version:7.10.0 ecs.version:1.5.0 event.action:logged-out event.category:authentication event.code:4634 event.created:Oct 5, 2021 @ 10:30:26.404 event.kind:event event.module:security event.outcome:success event.provider:Microsoft-Windows-Security-Auditing event.type:end host.name:Dcon3.uhtasi.local log.level:information message:An account was logged off. Subject: Security ID: S-1-5-21-3693063922-990322624-829951171-2256 Account Name: jumpsrv Account Domain: UHTASI Logon
`

[jertel]

There's not enough information to assist since you've omitted your rule configuration and doc sample set. I suggest enabling debug logging to watch the Elasticsearch queries and their responses to understand what is going on.

[roman-tasi]

I updated the post with what was missing

[jertel]

Moving to discussions since this does not qualify as an issue.

[jertel]

Since your query is a simple string containing a series of common words, and not a literal value to a specified field, most likely Elastic is matching on any record that contains any of the following words:

• A
• user
• account
• was
• locked
• out.

As you can imagine it is going to hit on just about every single record in Elasticsearch. I suggest reading up on how to format Lucene queries. Start here: https://elastalert2.readthedocs.io/en/latest/recipes/writing_filters.html

[roman-tasi]

I created a new filter, but am still having the similar result of an extremely high amount of matches and hits. Here is the new filter:

query : {

   constant_score : {
     filter : {
        bool : {
          must : [
            { term : {message : "A user account was locked out."}}
		]
     	}
     }
 }

}

(sorry some of the formatting got a little messed up)

Can you tell me what is wrong with my filter or give a filter that would work for me?

[jertel]

Include the full revised rule config, using triple backticks this time.

[roman-tasi]

  type: frequency

  num_events: 1

  timeframe:
    minutes: 60

 query : {

   constant_score : {
     filter : {
        bool : {
          must : [
            { term : {message : "A user account was locked out."}}
		]
     	}
     }
 }
}

  index: winlogbeat-*

  query_key:
    - source.ip

  include:
    - host.hostname
    - user.name
    - source.ip

  include_match_in_root: true

  alert_subject: "SSH abuse on <{}> | <{}|Show Dashboard>"
  alert_subject_args:
    - host.hostname
    - kibana_link

  alert_text: |-
    An attack on {} is detected.
    The attacker looks like:
    User: {}
    IP: {}
  alert_text_args:
    - host.hostname
    - user.name
    - source.ip

  alert_subject: "Lockout Alert"

  alert:
  - "email"
  email:
  - "info@hotmail.com"

  use_kibana4_dashboard: "https://dev.somewhere/app/kibana#/dashboard/37739d80-a95c-11e9-b5ba-33a34ca252fb"

[jertel]

Triple backticks are ``` not '''. I fixed it.

Your configuration is invalid. You have attempted to specify a top-level query parameter. But a query is a child of a filter parameter. Both the ElastAlert 2 documentation and the included examples, in examples/rules directory, show the correct syntax.