ElastiFlow v4.0.1

ElastiFlow v4.0.1 is a minor release. No migration of data from v4.0.0 to v4.0.1 is required.

WARNING! - If you are using a 3.x or earlier release, please refer to the v4.0.0 Breaking Changes.

Updates

• Update IP reputation dictionary

Fixes

• Netflow v5 sources reporting zero bytes and packets in ECS fields has been fixed.
• TSVB visualizations displaying data in bits/s now use the new bitd custom formatter.

ElastiFlow v4.0.0

WARNING! - ElastiFlow v4.0.0 is a major release, and now supports Elastic Common Schema (ECS). Due to significant data model changes there is no upgrade/migration from ElastiFlow 3.x. You should either remove all 3.x indices or deploy ElastiFlow 4.0.0 to a separate environment.

Breaking Changes

ElastiFlow v4.0.0 is built for Elasticsearch and Kibana 7.8.1 and later. No earlier versions will be supported. Please use a prior ElastiFlow release if you cannot yet upgrade to Elastic Stack 7.8.1+.

ElastiFlow v4.0.0 takes advantage of X-Pack Basic features, such as the Maps, SIEM and Logs apps, as well as Index Lifecycle Management (ILM). This means that you must use at least the X-Pack Basic licensed release of the Elastic Stack. The pure Apache 2.0 licensed release of the Elastic Stack will not work without disabling many features.

New Features

• Data model has changed to leverage ECS 1.5.
• Flow data can now be analyzed using the Kibana SIEM and Log apps.
• Optional resolution of MAC OUIs to vendor names (disabled by default).
• Kibana dark theme is now supported.
• Geo IP dashboards now leverage the new Kibana Maps app.
• Applications can now be defined manually by IP address and port number.
• Palo Alto virtual interface indexes are translated to interfaces names.
• Support for VeloCloud, Calix and various Cisco SD-WAN information elements.
• KQL is now default

Updates

• Pipeline refactored to simplify various logic, which might improve performance and throughput for some users.
• YAML dictionaries intended for customization by users have been moved to the logstash/elastiflow/user_settings path.

Fixes

• Client/Server detection using TCP flags is improved.

ElastiFlow v4.0.0-beta1

v4.0.0 is a major release. A data migration will be required if you want to have your older data available in 4.0.0. This BETA release does not yet include a migration method and is intended for testing with new flow data only.

Breaking Changes

ElastiFlow v4.0.0 is built for Elasticsearch and Kibana 7.5.0 and later. No earlier versions will be supported. Please use a prior ElastiFlow release if you cannot yet upgrade to Elastic Stack 7.5.x.

ElasiFlow v4.0.0 takes advantage of X-Pack Basic features, such as the Maps, SIEM and Logs apps, as well as Index Lifecycle Management (ILM). This means that you must use at least the X-Pack Basic licensed release of the Elastic Stack. The pure Apache 2.0 licensed release of the Elastic Stack will not work without disabling many features.

New Features

• Data model has changed to leverage ECS 1.4.
• Flow data can now be analyzed using the Kibana SIEM and Log apps.
• Optional resolution of MAC OUIs to vendor names (disabled by default).
• Kibana dark theme is now supported.
• Geo IP dashboards now leveage the new Kibana Maps app.
• Applications can now be defined manually by IP address and port number.
• Palo Alto virtual interface indexes are translated to interfaces names.

Updates

• Pipeline refactored to simplify various logic, which might improve performance and throughput for some users.
• YAML dictionaries intended for customization by users have been moved to the logstash/elastiflow/user_settings path.

ElastiFlow v3.5.3

v3.5.3 is a minor release. No migration of data from v3.5.x to v3.5.3 is required.

Breaking Changes

ElastiFlow v3.5.x provides support Elastic Stack 7.x. The support for document types has been completely removed in Elasticsearch 7.0.0. This has required changes to the index templates provided with ElastiFlow. You MUST first successfully upgrade to Elastic Stack 7.x PRIOR to using ElastiFlow v3.5.3.

New Features

• Added support for pmacct IEs (needed for VyOS 1.2.x).

ElastiFlow v3.5.2

v3.5.2 is a minor release. No migration of data from v3.5.1 to v3.5.2 is required.

Breaking Changes

ElastiFlow v3.5.x provides support Elastic Stack 7.x. The support for document types has been completely removed in Elasticsearch 7.0.0. This has required changes to the index templates provided with ElastiFlow. You MUST first successfully upgrade to Elastic Stack 7.x PRIOR to using ElastiFlow v3.5.2.

New Features

• Added normalization of WiFi-related Netflow v9 and IPFIX fields.
• The hostname where Logstash is running is provided in the field logstash_host.
• Added the ability to manually set flow sampling values for IPFIX.

Fixes

• Fix Cisco vzFlow type for list fields.
• Fix Procera IEs incorrectly defined as int.

Updates

• Improved the display of rate values in Vega visualizations.
• Added a lot of new Fortinet application IDs.
• Update IP reputation dictionary and GeoIP DBs.

ElastiFlow v3.5.1

v3.5.1 is a minor release. No migration of data from v3.5.0 to v3.5.1 is required.

Breaking Changes

ElastiFlow v3.5.x provides support Elastic Stack 7.x. The support for document types has been completely removed in Elasticsearch 7.0.0. This has required changes to the index templates provided with ElastiFlow. You MUST first successfully upgrade to Elastic Stack 7.x PRIOR to using ElastiFlow v3.5.1.

Fixes

• Updated environment variables in docker-compose.yml, which prevented the Kibana container from connecting to Elasticsearch.

Updates

• Update IP reputation dictionary and GeoIP DBs
• Minor updates to README.md, CHANGELOG.md and DOCKER.md

ElastiFlow v3.5.0

v3.5.0 is a minor release. No migration of data from v3.4.x to v3.5.0 is required.

Breaking Changes

ElastiFlow v3.5.0 provides support Elastic Stack 7.0.0. The support for document types has been completely removed in Elasticsearch 7.0.0. This has required changes to the index templates provided with ElastiFlow. You MUST first successfully upgrade to Elastic Stack 7.0.x PRIOR to using ElastiFlow v3.5.0.

New Features

• Support for Elastic Stack 7.0.x

Updates

• Dashboard tweaks for Kibana 7.0.x.

ElastiFlow v3.4.2

v3.4.2 is a minor release. No migration of data from v3.4.1 to v3.4.2 is required.

Breaking Changes

If you are upgrading from a release prior to 3.4.0, see the Breaking Changes notice for v3.4.0 below.

New Features

• Added support for Cisco AVC flow records (normalized to ElastiFlow schema)
• Determine client/server based on SYN+RST TCP flags
• Support for Elastic Stack 6.7.x

Updates

• Added A LOT of new Fortinet App IDs
• Index Pattern now includes all fields from codec definitions
• Updated GeoLite2-City and GeoLite2-ASN DBs
• Updated IP Reputation dictionary

Fixes

• Numerous index template fixes
• Removed duplicate TCP service names
• Fixed instances of double close brackets

ElastiFlow v3.4.1

v3.4.1 is a minor release. No migration of data from v3.4.0 to v3.4.1 is required.

Breaking Changes

If you are upgrading from a release prior to 3.4.0, see the Breaking Changes notice for v3.4.0.

New Features

• Added Docker support

Updates

• Updated GeoLite2-City and GeoLite2-ASN DBs
• Updated IP Reputation dictionary

Fixes

• Netflow application id regression. Now uses field netflow.app_id.

ElastiFlow v3.4.0

Breaking Changes

v3.4.0 adds custom field definitions for the Netflow codec. While greatly expanding the number of supported vendor-specific fields, many existing vendor-specific fields have been renamed. The ElastiFlow dashboards in previous releases were based on its normalized flow schema, or other standard Netflow and IPFIX fields, all of which are unchanged. However it may be necessary to update any Dashboards you created for the old vendor-specific field names to use the new names.

New Features

• Add a new Threats dashboard, based on IP reputation tags
• Netflow and IPFIX now default to included field definitions
• Provide a sysctl.d file to set net.core.rmem_max
• Added application ID support for Sophos, Sonicwall, Citrix Netscaler, IXIA IxFlow and Palo Alto
• Added support for Ziften ZFlow IPFIX host agents
• Added enrichment of enumerated values for many vendor-specific fields.

Updates

• Updated GeoLite2-City and GeoLite2-ASN DBs
• Updated IP Reputation dictionary
• Set all translate filters to use the new option refresh_behaviour, setting it to replace
• Updated FortiOS 5.6 Application IDs
• Disabled name lookups for connections to the tcp input
• Kibana index pattern now contains many new vendor-specific fields