Releases: airbnb/streamalert
v3.0.0
StreamAlert Release v3.0.0
New Features
LookupTables v2
In StreamAlert version 2, we added undocumented support for utilizing files stored in S3 as a method to dynamically load information for use within rules. This release completely redesigns LookupTables and adds new support for DynamoDB as a backend storage mechanism. The big driver behind this decision was the need to alleviate the high memory consumption that came with reading and deserializing JSON files from S3, among other benefits. Additionally, the new implementation will enable easily extending LookupTables to support other, arbitrary backend types.
Terraform Remote State File Locking
Given that StreamAlert utilizes a shared, remote state file that is stored in S3, it was effectively possible for multiple users to be reading and writing a state file at the same time. This could lead to state file corruption, and the potential to botch your entire deployment. In this release, we have added support for remote state file locking, via DynamoDB, to ensure that this type of corruption never occurs.
Improvements
Python 3.7 Support
Perhaps the most important part of this release is the full conversion of the codebase to Python 3.7. This has been a long time coming and we know that many have been patiently awaiting this. The Getting Started instructions have been updated to be specific to Python 3, and there is no backward compatibility to Python 2.7. See the note below on migration. A huge thanks to @GarretReece and @btonic for their support in this endeavor.
See also: #986
Terraform 0.12 Support
The release of the new version of Terraform introduced support for proper for each loops and "dynamic" blocks, along with numerous other benefits. These features have already enabled us to fix longstanding bug in our S3 event notifications module, and we will continue to leverage the benefits of Terraform version 12 to improve our Terraform modules going forward. A huge thanks to @scoders for their support in making this possible.
Separated Schema Files
This release introduces the ability to break up your logs.json file into multiple different files, nested within a schemas directory. It is now possible to have, for example, definition files of schemas/osquery.json and/or schemas/aws.json to independently house all of your osquery and aws related log definitions, respectively. This feature is currently backwards compatible, so if your logs.json is not unmanageably large, or you are not ready to make the change, then your current definitions will continue to function just fine.
Moved Sources into Cluster Files
Version 3 of StreamAlert includes the migration of the cluster source definitions to reside within the individual cluster configuration files. Be sure to check the new format in the StreamAlert docs on the Datasource Configuration page and update your deployment configs accordingly.
To view the complete list of all of the improvements in v3.0.0, including many not mentioned above, see here.
Notable Fixes
AWS Resource Prefix Consistency
This release includes sweeping changes to correct any and all discrepancies related to prefixing or namespacing with the streamlert name. All IAM roles are now created under a valid namespace, each Kinesis Data Firehose is now prefixed properly, and even the pesky “stream_alert” naming convention has been thoroughly scrubbed in this update. We know that these changes may lead to many complications with upgrading, so please see the note below on migration.
Alert Merger Optimization
The alert merger function has been updated to better handle massive alert throughput, reducing the potential for out-of-memory exceptions in the function.
Terraform Refactors
Various Terraform modules have also been refactored to reduce complexity or redundancy.
StreamAlert Apps Updates
The G Suite StreamAlert Apps have been updated to support newer versions of the Python client library for Google. Additionally, the Box StreamAlert App dependencies have been updated to an official, non-alpha release that supports JSON Web Tokens (JWT) for authentication.
Bug Fixes
To view the complete list of all of the bugs fixed in v3.0.0, including many not mentioned above, see here.
All Changes
To view the complete list of all changes included in v3.0.0, see here.
Migration Steps
There are many changes in this release that will make it prohibitively difficult to do a direct "upgrade" from version 2 to version 3. Therefore, we recommend updating your prefix setting and doing a complete redeploy of StreamAlert version 3. This should be possible alongside a current deployment, thanks in part to the changes that are included in release 3.
After deploying version 3, you must also migrate over your data sources/producers as you see fit. This step is beyond the scope of the advice we're willing to provide here, but please reach out to us if you are stuck and need assistance.
At a minimum, any rules will need to be updated for new import paths (stream_alert --> streamalert) along with any Python 2 vs Python 3 syntax changes. We recommend using the Python 2to3 tool on all custom rules to begin your conversion, which should address most of the necessary changes.
Thanks
We would also like to give a special shout out to @jack1902 and @0xdabbad00 for their contributions to version 3.
v2.3.0
v2.2.1
A complete list of changes in v2.2.1 can be found here
Highlighted Changes
- Deployments of streamalert now require a certain configuration variable to specify SQS prefix names. This is not a breaking change, but will require specifying the configuration or Terraform builds will not work. Refer to this PR (#960) for more details.