Fluentbit forwading logs to ElasticSearch with timestamps in the future

[ricsanfre]

All my homelab RaspberryPI nodes are configured in Europe/Madrid Timezone: gateway and k3s nodes
For k3s cluster nodes I am using Fluentd deployed as daemonset for parsing node logs (/var/log/syslog and /var/log/auth.log) without issues of generating logs in the future. Fluentd parsing configuration enables the definition of Local Time Zone

But for gateway node I am using fluentbit as lightweight version of fluentd. Fluentbit when parsing local syslog and auth.log files is parsing the events extracting properly the timestamp but they are stored in elasticsearch in the future (with timestamps that are 2 hours ahead of the actual timestmap).

There is an issue with Fluetbit which is assuming UTC timezone for all logs. There is parser configuration option Time_Offset available to resolve this, but it allows only to configure fixed UTC offsets not taking into account variable offsets. In my local time zone because of the summer daylight saving, there is UTC+1 offset in Winter and UTC+2 offset in Summer.

See open issue in fluentbit git repo fluent/fluent-bit#593

[ricsanfre]

For solving the issue the approach described in fluent/fluent-bit#593 will be used: Configure fluentbit with a filter modifying the timestamp of the syslog messages before forwarding them to Elasticsearch. The Filter will convert timestamp field to UTC using a a LUA script.

This is the filter that can be included in fluentbit configuration

[FILTER]
    Name         lua
    Match        *
    script       /fluent-bit/etc/adjust_ts.lua
    call         local_timestamp_to_UTC

/fluent-bit/etc/adjust_ts.lua

function cb_fix_timestamp(tag, timestamp, record)
    local utcdate   = os.date("!*t", ts)
    local localdate = os.date("*t", ts)
    localdate.isdst = false -- this is the trick
    utc_time_diff = os.difftime(os.time(localdate), os.time(utcdate))
    return 1, timestamp - utc_time_diff, record
end

[ricsanfre]

Fluentbit ansible role modified to support lua scripts (v.1.0.3)