Processor/logratelimitprocessor

[anuraj381]

Description

This PR add a new Processor for rate-limiting logs based on several configurable log fields.
The logratelimit processor is useful in the situation where you are collecting logs from multiple services on every otel-collector pod and there are one or many services which are logging too many logs such that it increases noise in the entire pipeline and storage (otel backend). So to control the log noise/surge from any service/namespace/pod etc. you can use the rate-limit processor, you can give the fields on which you want to limit logs and add config for allowed rate and interval.

Link to tracking issue

#35204

Testing

Tested with some real data in development environment, unit tests to be written in next PR

Documentation

Added code documentation and added a Readme file with processor details and config info / how to use

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: anuraj381 / name: Anuraj Jain (612a8ec, 50ff53d, 3856a59, e0a8430, a176806, a86db14, b6ddea1, 1857fc5)

[bogdandrutu]

Confused that in the issue we are talking about a generic rate limit vs a logs specific one.

[anuraj381]

@bogdandrutu the discussion in the issue mentioned is entirely focused on logs rate-limiting which is actually the pain point as logs in high scale production system often get noisy from some services/contributors and there is currently no solution in otel-collectors to rate-limit them, however for traces solution is already available in tail-sampling processor to rate-limit traces/spans.
However if in-case this processor has to be generic rate-limiter then it can be extended to accommodate traces and metrics in later PRs.

[atoulme]

You have not secured a sponsor for this new component. Please attend a SIG meeting and make your case. Alternatively, please consider hosting your component in a separate repository. See https://github.com/otel-warez/stdinreceiver for an example.

[anuraj381]

@atoulme @mx-psi can we please add this to discussion agenda for next SIG meeting. Also I see next SIG meeting for collector implementation on 18th Dec, 24 6:30AM IST, please let me know if this is the right one, I will attend, checked from here - https://github.com/open-telemetry/community?tab=readme-ov-file#implementation-sigs

[atoulme]

@atoulme @mx-psi can we please add this to discussion agenda for next SIG meeting. Also I see next SIG meeting for collector implementation on 18th Dec, 24 6:30AM IST, please let me know if this is the right one, I will attend, checked from here - https://github.com/open-telemetry/community?tab=readme-ov-file#implementation-sigs

yes this is the next SIG meeting, in 90 minutes or so. You can add any agenda item to the meeting, I can do it for you this time around.

[MovieStoreGuy]

Reminds me of the conversation #6908