Support encryption in Fluent-Forward

[ScarletTanager]

Currently Fluent-Forward supports data compression via gzip when using CompressedPackedForward mode. We see a lot of demand from customers/users for securing the contents of log messages during transit/processing, and this extends to use cases (e.g. handling in a multi-tenant backplane) where e.g. TLS is not sufficient/does not necessarily apply. I would suggest that we might extend the FF protocol to include support for encryption.

Possible approach: Perhaps extend the range of allowed options in the options hash to include something like:

{"encrypted":"aes256"}

Where the key is encrypted, and the value is the specific algorithm/key length used. In a single-tenant or similar installation, the key could be supplied as part of the endpoint configuration (fluent-bit, fluentd, or another implementation). In a multi-tenant environment, where different messages might be encrypted using tenant-specific keys, the option might take the form of something like:

{"encrypted":{"cipher":"aes256","key_id":"XXXYYYZZZ"}}

where XXXYYYZZZ is an implementation-specific identifier (e.g. a GUID) used to lookup and retrieve the correct encryption key (e.g. from some sort of key-wrapping solution, such as IBM KeyProtect or similar offerings from other vendors).

An alternative we have considered is extending the protocol on our own and implementing it in our own custom output/input plugins, but we think there is sufficient value to the idea to warrant inclusion in the base wire protocol.

Questions:

1. Is this the simplest way to address this?
2. Is this applicable to other scenarios/users/vendors?
3. Are there shortcomings/negative side-effects?

My organization would be very interested in a feature like this and would be willing to contribute to discussion/design/implementation (within fluent-bit), if this were to move forward.