pfsense interface and gateway metrics to pfelk - working solution

[robeweber]

I was very much interrested in also getting regular updates on the state of all used interfaces from pfSense on pfELK.
VictorRobellini's telegraf_pfifgw.php script was my starting point.
However I did not want to use telegram and I did not want it to output influx formatted output.

So I changed the PHP code to be used by the cron plugin of pfSense (install this package if you want to also use this).
Download the newly created PHP script:
(https://github.com/robeweber/pfelk/blob/main/pfsense_interfaces.php)

In pfSense, upload the script using the "Upload File" section:

pfSense will display the path where it has uploaded the file to.
It will most likely be /tmp/pfsense_interfaces.php

Using the "Execute Shell Command" on the same page of pfSense, execute the following code to move it to the right place:

mv /mnt/pfsense_interfaces.php /usr/local/bin/pfsense_interfaces.php

I have set the timing to the maximum possible I could figure out:

Set Command to the following:
/usr/local/bin/php-cgi -f /usr/local/bin/pfsense_interfaces.php | /usr/bin/nc YOURPFELKIP 5057

Set logstash to also listen on port 5057. For that I have written a new conf.d pfelk file:
03-input-pfsense-metrics.pfelk

# 03-inputs.pfsense
#
input {
  ## pfsense metrics ##
  syslog {
    id => "pfelk-pfsense-0001"
    type => "firewall"
    port => 5057
    syslog_field => "message"
    ecs_compatibility => v1
    grok_pattern => "<%{POSINT:[log][syslog][priority]}>%{GREEDYDATA:pfelk}"
    tags => ["pfelk"]
  }
}
# 
filter {
  grok {
    patterns_dir => [ "/etc/pfelk/patterns" ]
    match => [ "pfelk", "%{PFSMETRICS}" ]
  }
#### RFC 5424 Date/Time Format ####
  date {
    match => [ "[event][created]", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss", "ISO8601" ]
    target => "[event][created]"
  }
}

Add the GROK patterns in patterns/pfelk.grok:

# pfSense - Interfaces
PFSMETRICS %{SYSLOGTIMESTAMP:[event][created]}\s%{SYSLOGHOST:[log][syslog][hostname]}\s%{PROG:[log][syslog][appname]}\:\s%{GREEDYDATA:filter_message}\\n
PFINTERFACE %{WORD:[interface][name]},%{IPORHOST:[interface][ipv4]},%{INT:[interface][subnet]},%{IPORHOST:[interface][ipv6]},%{IPORHOST:[interface][sugnetv6]},(%{MAC:[interface][mac]}|%{WORD:[interface][mac]}),%{DATA:[interface][friendlyname]},%{WORD:[interface][source]}.%{GREEDYDATA:[interface][status]}
PFGATEWAY %{WORD:[gateway][interface]},%{DATA:[gateway][name]}\s\"%{IPORHOST:[gateway][monitor]}\",\"%{IPORHOST:[gateway][source]}\",%{INT:[gateway][defaultgw]},\"%{DATA:[gateway][description]}\",%{NUMBER:[gateway][delay]},%{NUMBER:[gateway][stddev]},%{NUMBER:[gateway][loss]},\"%{WORD:[gateway][status]}\",\"%{WORD:[gateway][substatus]}\"

Add the following two new sections in:
conf.d/05-apps.pfelk

  ### pfSense Interfaces
  if [log][syslog][appname] =~ /^interfaces/ {
    mutate {
      add_tag => "pfSenseInterfaces"
    }
    grok {
      patterns_dir => [ "/etc/pfelk/patterns" ]
      match => [ "filter_message", "%{PFINTERFACE}" ]
    }
  }
  ### pfSense Gateways
  if [log][syslog][appname] =~ /^gateways/ {
    mutate {
      add_tag => "pfSenseGateways"
    }
    grok {
      patterns_dir => [ "/etc/pfelk/patterns" ]
      match => [ "filter_message", "%{PFGATEWAY}" ]
    }
  }

And add the two following "else if" statements just before the "else" statement in:

50-outputs.pfelk
  } else if [log][syslog][appname] == "interfaces" {
    mutate { add_field => { "[data_stream][namespace]" => "pfinterfaces" } }
  } else if [log][syslog][appname] == "gateways" {
    mutate { add_field => { "[data_stream][namespace]" => "pfgateways" } }

in the Kibana dashboard, after the first data has been sent to elasticsearch, you will have to create two new "Data Views" in order to see the data in the Discover tab:

*-pfelk-pfinterfaces*
*-pfelk-pfgateways*

And you are done and enjoy.

I have now uploaded all files as I am using them at this moment.
https://github.com/robeweber/pfelk/tree/main
Only download and overwrite them in your setup if you have not changed anything else yourselves in them and only if you follow all the steps of (#486) and #487