v2.5.2

7th February 2021

Log Courier / Log Carver

• Fix integer overflow preventing 32-bit compilation

v2.5.1

7th February 2021

Log Courier

• Restore backwards compatibility with v2.0.6 by removing ECS fields that were added in v2.5.0
• New configuration option to enable ECS fields: enable ecs
• ECS fields now also obey the corresponding add xxx configuration options

Log Courier / Log Carver

• Fix vendoring issue of a dependency that was blocking package builds
• No binary changes to Log Courier / Log Carver

v2.5.0

2.5.0

Log Courier

• Fix a few issues that could cause Log Courier to hang unnecessarily during shutdown
• Improvements to configuration parsing
• Optimised registrar to only write periodically and not constantly
• Buffers and compression streams are reused during event transmission to further reduce memory and garbage collection
• Rebuilt the transport layer to use less routines per connection and negotiate a new method of event transmission that uses less memory
• Rebuilt application initialisation to allow creation of separate binaries for different tasks running under the same pipelining principle and using similar transports (e.g. Log Carver)
• Add option to enable ECS (Elastic Common Schema) for the builtin fields such as host and file path
• Many more under-the-hood changes to make code more straight forward and to allow code sharing with Log Carver

Log Carver

• Initial beta release
• Can be used as a low memory/CPU substitute to Logstash for basic events
• Supported processor actions include: date, geoip, user_agent, kv, add_tag, remove_tag, set_field, unset_field
• The set_field action supports the Common Expression Language (CEL) for code-like expression support when setting fields
• If/ElseIf/Else support in the pipeline using Common Expression Language (CEL) for code-like conditional expressions
• Receives events over the Log Courier protocol from Log Courier clients
• New ES transport to allow events to be sent directly to Elasticsearch
• Templates embedded for ES6+ that will automatically be inserted as "logstash"
• It is recommended to use new indices and remove the "logstash" template from ES as fields are different and now follow ECS (Elastic Common Schema)
• Configuration documentation is minimal but a minimal example can be found in the docs/examples folder, more will be added in time

v2.0.6

9th May 2020

• Fix several issues when shutting down with outstanding payloads, to prevent hanging forever

v2.0.5

18th February 2017

• Fix -stdin run mode not being able to exit cleanly (#353)
• Fix address pool rotation so that the same connection address is used on every
  connection attempt
• Fix certificate problem notices showing invalid year values
• Fix end-to-end tests

View the full Change Log.

v2.0.4

11th June 2016

Log Courier

• Fix random recovery failure when Logstash is unavailable (#324)
• Fix systemd unit files User= declaration that does not support variables. In
  order to change the user that Log Courier runs as, edit the unit file directly
  after updating to 2.0.4 (#322)
• Fix max pending payloads exceeded when configuration is reloaded or when there
  are multiple Logstash connections and one of the connections recovers
• Fix service start failure after a reboot on systems where /var/run is tmpfs
  (#321)
• Fix shipping encoding error when configuration file is YAML and global fields
  or per-file declaration fields contain nested maps (#325)
• Add the ability to set the group that log-courier runs as

RPM Packaging

• Removed the dependency on ZeroMQ 3 as this was deprecated in 2.x of Log
  Courier

View the full Change Log.

v2.0.3

9th May 2016

• Fix issue where pending payload limit is exceeded when Logstash is overloaded
  (#315)
• Fix incorrect stale bytes count in API / lc-admin
• Fix the harvester status not updating correctly when reaching end of file in
  API / lc-admin
• Reintroduce full status command into lc-admin

View the full Change Log.

v2.0.2

8th May 2016

• Fix a rare hang when endpoint failure occurred (#314)
• Add a new debug command to the REST API and lc-admin tool which gives a
  live stack trace of the application (#315)

View the full Change Log.

v2.0.1

25th April 2016

• Fix lc-admin ignoring -config parameter and not auto-loading the default
  configuration in RPM and DEB packages (#303)
• Fix lc-admin numerical outputs such as file completion percentage showing as
  1.00e2 instead of 100 (#304)
• Fix hang that could occur when Logstash failed in loadbalance or failover
  network methods (#311)
• Fix admin listen address not having a default (#307)
• Improve documentation for lc-admin (#307)
• Improve debug logging of backoff calculations

View the full Change Log.

v2.0.0

20th April 2016

Log Courier 2.x is compatible with the 1.x Logstash plugins.

_Breaking Changes_

• CurveZMQ transport has been removed
• The lc-curvekey utility has been removed
• Multiline codecs can now be configured with multiple patterns. As such, the
  pattern configuration has been replaced with patterns and it is now an array
  of patterns
• Multiline codecs have a new match configuration that can be set to all or
  any, to control how multiple patterns are used. The default is any.
• Multiline patterns can be individually negated with a "!" prefix. The negate
  configuration directive has been removed. A "=" prefix is also possible to allow
  patterns that need to start with a literal "!"
• Filter codec patterns now also accept negation in the same method. The
  negate configuration directive has also been removed for filters and a new
  match configuration added that also defaults to any
• Multiple codecs can now be specified. As such, the codec configuration has
  been renamed to codecs and must now be an array.
• The persist directory configuration is now required, unless it was built in
  at build time (which it will be for RPM and DEB packages - see Build Changes).
• All admin prefixed configurations have been moved from the general section
  into their own admin section, and the prefix removed
• The reconnect transport configuration has been removed, and replaced with
  reconnect backoff and reconnect backoff max

_Changes_

• A new global fields configuration is available in the general section
  where fields that are to be added to all events from all paths can be specified.
  This complements the current fields configuration that is per-path.
• A new method configuration directive has been added to the network
  section, which allows random, failover or loadbalance network modes.
  Information on these new values can be found in the configuration documentation.
  The default is random for backwards compatibility.
• The dead time stream configuration directive now defaults to 1 hour
• New backoff configurations, failure backoff and failure backoff max, have
  been added, to allow backoff from problematic remote endpoints where, for
  example, connection always succeeds but transmission attempts always timeout.
• Configuration files can now be in YAML format by giving the file a .yaml
  extension, and this will be the preferred format. JSON format will continue to
  be used for .conf and .json configuration files
• The config parameter can now be omitted from log-courier if a default
  configuration file was specified during build (see Build Changes)
• lc-admin can now be given a configuration file to load the
  admin.listen address from using the config parameter. In the absense of
  both connect and config parameters it will load the default configuration file if
  one was specified during build (see Build Changes)
• Log Courier remote administration via lc-admin is now a REST interface,
  allowing third-party integrations and monitoring
• Harvester snapshot information reported by lc-admin now contains additional
  information such as the last known file size and percentage completion (#239)
• Harvester snapshot information is now updated even when the remote server is
  down
• Publisher snapshot information reported by lc-admin now contains a list of
  enabled endpoints and their statuses (#199)
• If the harvester detects a log file has had data written without a new line
  ending, it will report a warning to the log so it can be investigated

_Fixes_

• Do not open dead files on startup (which causes excessive memory usage if
  there are many of them) if their size has not changed since they were last
  opened (#242)
• When a long line was split, the "splitline" tag was not being appended if the
  fields configuration already contained tags

_RPM Packaging_

• Fix broken systemd configuration (#237)
• Fix non-fatal %postun script failure, and fix starting of Log Courier during
  package upgrade even if it was not originally started (#228)

_DEB Packaging_

• Fix broken systemd configuration (#237)

_Build Changes_

• Some configuration parameters, such as default configuration file and default
  persist directory can now be specified at build time by setting environment
  variables for them. For example: LC_DEFAULT_CONFIGURATION_FILE and
  LC_DEFAULT_GENERAL_PERSIST_DIR. These will be used in the RPM and DEB packages
  to set platform specific defaults
• Simplified the entire build process to use only Go tooling to make
  cross-platform building significantly easier, especially on Windows
• Discarded Ruby test framework in favour of developing a Go test framework over
  time

View the full Change Log.