using pfelk 22 with index instead of Data stream #408

edrisvton · 2022-02-13T14:10:04Z

edrisvton
Feb 13, 2022

hello every one
before last update im using pfelk Version 8 with pfsense 2.5 and every thing work fine in indices i have pfelk-firewall-yyyy-mm im trying to use pfelk 22 but i need too have indices instead of data stream because the time series showed in indices good for managment and backup and want to have my logs per month but when i implement with data stream all thing show is just one data steam
i tried to implement pfelk 22 with elasticsearch and /etc/pfelk/conf.d/* pfelk 21 (too use last change with index template model) but i didn't work at all
error.pfelk.log
im not advance in ELK but can someone help me to using pfelk 22 with index model not data stream (need too backup montly my log to another server )
thanks to @a3ilson for this wonderful pfelk

Answered by a3ilson

Feb 13, 2022

@edrisvton

I would suggest the following two options but would recommend datastream over the indices

Reference: https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams.html

ILM

You are able to automate the management via the pfelk ILM policy which may be modified to your needs, amended or a separate ILM created.

Indicies (Legacy)

The older 50-outputs.conf (non-data stream) should still work:

# 50-outputs.conf
################################################################################
# Version: 22.01                                                               #
# Required: Yes                                                                #
# Description: Sen…

View full answer

a3ilson · 2022-02-13T15:40:55Z

a3ilson
Feb 13, 2022
Maintainer

@edrisvton

I would suggest the following two options but would recommend datastream over the indices

Reference: https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams.html

ILM

You are able to automate the management via the pfelk ILM policy which may be modified to your needs, amended or a separate ILM created.

Indicies (Legacy)

The older 50-outputs.conf (non-data stream) should still work:

# 50-outputs.conf
################################################################################
# Version: 22.01                                                               #
# Required: Yes                                                                #
# Description: Sends enriched logs to Elasticsearch. Remove "#ILM#" to enable  #
# ILM. ILM broken...pending datastream implementation.                         #
################################################################################
#
filter {
  if [process][name] == "captiveportal" {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-captiveportal-%{+YYYY.MM.dd}" } }
  } else if [process][name] == "dhcp" {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-dhcp-%{+YYYY.MM.dd}" } }
  } else if [process][name] == "firewall" {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-firewall-%{+YYYY.MM.dd}" } }
  } else if "bind9" in [tags] or "dpinger" in [tags] or "ntpd" in [tags] or "web_portal" in [tags] {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-firewall-processes-%{+YYYY.MM.dd}" } }
  } else if [process][name] == "haproxy" {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-haproxy-%{+YYYY.MM.dd}" } }
  } else if [process][name] == "openvpn" {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-openvpn-%{+YYYY.MM.dd}" } }
  } else if [process][name] == "unbound" {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-unbound-%{+YYYY.MM.dd}" } }
  } else if [process][name] == "suricata" {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-suricata-%{+YYYY.MM.dd}" } }
  } else if [process][name] == "snort" {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-snort-%{+YYYY.MM.dd}" } }
  } else if [process][name] == "squid" {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-squid-%{+YYYY.MM.dd}" } }
  } else {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-unknown-%{+YYYY}" } }
  }
}
output {
  if [type] == "beats" {
    elasticsearch {
      hosts => ["http://localhost:9200"]
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      ecs_compatibility => "v1"
      # manage_template => false
      ### X-Pack Username and Password ###
      # user => USERNAMEHERE
      # password => PASSWORDHERE
    }
  }
  elasticsearch {
    # index => "%{[@metadata][target_index]}"
    ecs_compatibility => "v1"
    ### X-Pack Username and Password ###
    # user => USERNAMEHERE
    # password => PASSWORDHERE
  }
}

1 reply

edrisvton Feb 14, 2022
Author

@edrisvton

I would suggest the following two options but would recommend datastream over the indices

Reference: https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams.html

ILM

You are able to automate the management via the pfelk ILM policy which may be modified to your needs, amended or a separate ILM created.

Indicies (Legacy)

The older 50-outputs.conf (non-data stream) should still work:

# 50-outputs.conf
################################################################################
# Version: 22.01                                                               #
# Required: Yes                                                                #
# Description: Sends enriched logs to Elasticsearch. Remove "#ILM#" to enable  #
# ILM. ILM broken...pending datastream implementation.                         #
################################################################################
#
filter {
  if [process][name] == "captiveportal" {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-captiveportal-%{+YYYY.MM.dd}" } }
  } else if [process][name] == "dhcp" {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-dhcp-%{+YYYY.MM.dd}" } }
  } else if [process][name] == "firewall" {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-firewall-%{+YYYY.MM.dd}" } }
  } else if "bind9" in [tags] or "dpinger" in [tags] or "ntpd" in [tags] or "web_portal" in [tags] {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-firewall-processes-%{+YYYY.MM.dd}" } }
  } else if [process][name] == "haproxy" {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-haproxy-%{+YYYY.MM.dd}" } }
  } else if [process][name] == "openvpn" {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-openvpn-%{+YYYY.MM.dd}" } }
  } else if [process][name] == "unbound" {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-unbound-%{+YYYY.MM.dd}" } }
  } else if [process][name] == "suricata" {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-suricata-%{+YYYY.MM.dd}" } }
  } else if [process][name] == "snort" {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-snort-%{+YYYY.MM.dd}" } }
  } else if [process][name] == "squid" {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-squid-%{+YYYY.MM.dd}" } }
  } else {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-unknown-%{+YYYY}" } }
  }
}
output {
  if [type] == "beats" {
    elasticsearch {
      hosts => ["http://localhost:9200"]
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      ecs_compatibility => "v1"
      # manage_template => false
      ### X-Pack Username and Password ###
      # user => USERNAMEHERE
      # password => PASSWORDHERE
    }
  }
  elasticsearch {
    # index => "%{[@metadata][target_index]}"
    ecs_compatibility => "v1"
    ### X-Pack Username and Password ###
    # user => USERNAMEHERE
    # password => PASSWORDHERE
  }
}

thank you for your replay
still not working i change 50-output.conf and restart services (all configs is in 22 version) but noting revives and show in indices
my pfelk index template version is 9 is that correct ?? should i change to version 22 (in 22 version template has data stream option)

a3ilson · 2022-02-14T14:11:55Z

a3ilson
Feb 14, 2022
Maintainer

Give the 50-outputs.conf another try... I just realized the prior one was incorrect:

50-outputs.conf

# 50-outputs.conf
################################################################################
# Version: 22.01                                                               #
# Required: Yes                                                                #
# Description: Sends enriched logs to Elasticsearch. Remove "#ILM#" to enable  #
# ILM. ILM broken...pending datastream implementation.                         #
################################################################################
#
filter {
  if [process][name] == "captiveportal" {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-captiveportal-%{+YYYY.MM.dd}" } }
  } else if [process][name] == "dhcp" {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-dhcp-%{+YYYY.MM.dd}" } }
  } else if [process][name] == "firewall" {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-firewall-%{+YYYY.MM.dd}" } }
  } else if "bind9" in [tags] or "dpinger" in [tags] or "ntpd" in [tags] or "web_portal" in [tags] {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-firewall-processes-%{+YYYY.MM.dd}" } }
  } else if [process][name] == "haproxy" {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-haproxy-%{+YYYY.MM.dd}" } }
  } else if [process][name] == "openvpn" {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-openvpn-%{+YYYY.MM.dd}" } }
  } else if [process][name] == "unbound" {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-unbound-%{+YYYY.MM.dd}" } }
  } else if [process][name] == "suricata" {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-suricata-%{+YYYY.MM.dd}" } }
  } else if [process][name] == "snort" {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-snort-%{+YYYY.MM.dd}" } }
  } else if [process][name] == "squid" {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-squid-%{+YYYY.MM.dd}" } }
  } else {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-unknown-%{+YYYY}" } }
  }
}
output {
  if [type] == "beats" {
    elasticsearch {
      hosts => ["http://localhost:9200"]
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      # ecs_compatibility => "v1"
      # manage_template => false
      ### X-Pack Username and Password ###
      # user => USERNAMEHERE
      # password => PASSWORDHERE
    }
  }
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "%{[@metadata][target_index]}"
    manage_template => false
    # ecs_compatibility => "v1"
    ### X-Pack Username and Password ###
    # user => USERNAMEHERE
    # password => PASSWORDHERE
  }
}

Templates

Your templates should be fine but you can also update to the latest. However, you'll need to disable datastream if electing to utilize indices. This can be done by unticking the data stream for each:

3 replies

a3ilson Feb 16, 2022
Maintainer

@edrisvton - let me know if that worked out for you

edrisvton Feb 19, 2022
Author

@edrisvton - let me know if that worked out for you

thank you for answers but still not work
i thing it logstash problem because in tcpdump i can see syslog traffic on ELK server but logstash not receive i double check pipline config and in logstash error see this
ERROR][logstash.javapipeline ][pfelk] Pipeline error {:pipeline_id=>"pfelk", :exception=>#<Grok::PatternError: pattern
after this i check my /etc/pfelk/patterns/pfelk.grok and config was 21 version , i changed it to version 22 and reset logstash and problem solved
[INFO ][logstash.inputs.syslog ][pfelk][pfelk-firewall-0001] Starting syslog udp listener {:address=>"0.0.0.0:5140"}
[INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:pfelk], :non_running_pipelines=>

now every thing work but have some problem in discover show index and dashboard ( i think i can handle it !! :) )

thank you for reply

a3ilson Feb 19, 2022
Maintainer

That error was likely the result of not having the two grok patterns within the patterns folder. Try v22 and ensure you have all the files (there should be pfelk.grok and openvpn.grok within the patterns directory).

As for the indices, the last update for v21 (50-outputs.conf) also created firewall-processes and unknown indices. These are logs that (1) were not filterlog messages or (2) were not defined anywhere (unknown).

edrisvton · 2022-02-20T05:54:01Z

edrisvton
Feb 20, 2022
Author

thank you for replay
my Pfelk work fine but dashboards and visualize not show because of index pattern index pattern show in kibana is "pfelk-firewall*"
but for suricata is -pfelk-suricata .
im trying to add index pattern with format -pfelk-firewall but got error {The index pattern you entered doesn't match any data streams, indices, or index aliases. You can match 11 sources.} so Im forced to create index pattern with difference [pfelk-firewall*] working fine but all visualizes got below error

Could not locate that index-pattern (id: id-firewall), click here to re-create it
to resolve this we must re-create all visualizes (i just type it here for if someone got this error)

0 replies

edrisvton · 2022-02-20T06:07:36Z

edrisvton
Feb 20, 2022
Author

sorry but another question
indicies save logs in day format per index for example pfelk-firewall-2022.02.19-pfelk - firewall-2022.02.20 ....
where can i change it too index it monthly
and except of index lifecycle policies and snapshot is any way to export index (like download it) and save it in offline storage (like HDD) so any time we need that month load that log into ELK and browse it (I know it very perfectionist :/ )

0 replies

a3ilson · 2022-02-20T14:30:41Z

a3ilson
Feb 20, 2022
Maintainer

@edrisvton

Index Patterns

This is a result of utilizing the new v22+ saved objects which were changed to support the default data stream
Data Streams are required to be in a data_stream.namspace format (e.g. logs.pfelk.firewall)
[Solution] utilize the latest saved objects prior to v2. The only change to the dashboard was an amendment from the referenced index patterns (e.g. from pfelk-firewall to *-pfelk-firewall*)

Index Format

I would highly recommend ILM for this (automate based on time and/or size)
[Solution] However, to do what your asking for, amend the 50-outputs.conf as follows:

# 50-outputs.conf
################################################################################
# Version: 22.01                                                               #
# Required: Yes                                                                #
# Description: Sends enriched logs to Elasticsearch. Remove "#ILM#" to enable  #
# ILM. ILM broken...pending datastream implementation.                         #
################################################################################
#
filter {
  if [process][name] == "captiveportal" {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-captiveportal-%{+YYYY.MM}" } }
  } else if [process][name] == "dhcp" {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-dhcp-%{+YYYY.MM}" } }
  } else if [process][name] == "firewall" {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-firewall-%{+YYYY.MM}" } }
  } else if "bind9" in [tags] or "dpinger" in [tags] or "ntpd" in [tags] or "web_portal" in [tags] {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-firewall-processes-%{+YYYY.MM}" } }
  } else if [process][name] == "haproxy" {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-haproxy-%{+YYYY.MM}" } }
  } else if [process][name] == "openvpn" {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-openvpn-%{+YYYY.MM}" } }
  } else if [process][name] == "unbound" {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-unbound-%{+YYYY.MM}" } }
  } else if [process][name] == "suricata" {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-suricata-%{+YYYY.MM}" } }
  } else if [process][name] == "snort" {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-snort-%{+YYYY.MM}" } }
  } else if [process][name] == "squid" {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-squid-%{+YYYY.MM}" } }
  } else {
    mutate { add_field => { "[@metadata][target_index]" => "pfelk-unknown-%{+YYYY.MM}" } }
  }
}
output {
  if [type] == "beats" {
    elasticsearch {
      hosts => ["http://localhost:9200"]
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      # ecs_compatibility => "v1"
      # manage_template => false
      ### X-Pack Username and Password ###
      # user => USERNAMEHERE
      # password => PASSWORDHERE
    }
  }
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "%{[@metadata][target_index]}"
    manage_template => false
    # ecs_compatibility => "v1"
    ### X-Pack Username and Password ###
    # user => USERNAMEHERE
    # password => PASSWORDHERE
  }
}

Removing the 'dd' from +YYYY.MM.dd will result in new indices every month.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pfelk

using pfelk 22 with index instead of Data stream #408

{{title}}

Replies: 5 comments 4 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

ILM

Indicies (Legacy)

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

pfelk

using pfelk 22 with index instead of Data stream #408

edrisvton Feb 13, 2022

ILM

Indicies (Legacy)

Replies: 5 comments · 4 replies

a3ilson Feb 13, 2022 Maintainer

ILM

Indicies (Legacy)

edrisvton Feb 14, 2022 Author

ILM

Indicies (Legacy)

a3ilson Feb 14, 2022 Maintainer

50-outputs.conf

Templates

a3ilson Feb 16, 2022 Maintainer

edrisvton Feb 19, 2022 Author

a3ilson Feb 19, 2022 Maintainer

edrisvton Feb 20, 2022 Author

edrisvton Feb 20, 2022 Author

a3ilson Feb 20, 2022 Maintainer

Index Patterns

Index Format

edrisvton
Feb 13, 2022

Replies: 5 comments 4 replies

a3ilson
Feb 13, 2022
Maintainer

edrisvton Feb 14, 2022
Author

a3ilson
Feb 14, 2022
Maintainer

a3ilson Feb 16, 2022
Maintainer

edrisvton Feb 19, 2022
Author

a3ilson Feb 19, 2022
Maintainer

edrisvton
Feb 20, 2022
Author

edrisvton
Feb 20, 2022
Author

a3ilson
Feb 20, 2022
Maintainer