Improve unit testing capabilities of CEL-based inputs in Integrations

[tommyers-elastic]

We are building more and more complex CEL programs to handle inputs for API-based monitoring, e.g. Rubrik, MS365, OpenAI. However there is no way to run unit tests on the CEL code itself. I'm worried about maintainability and future stability of the growing CEL codebase in Integrations.

Ideally we should be able to define API responses and properly unit test the CEL code without the overhead of running the filebeat input against the real API. I'm thinking like the existing pipeline tests, but for the CEL programs in Integration packages.

[tommyers-elastic]

@efd6 @andrewkroh

pinging you here because i know you are working on some CEL validation related stuff. any thoughts on how this might work?

[efd6]

We use github.com/elastic/stream to define mocked APIs and test against those. You can see this in the majority of CEL-based integrations, but for example in abnormal_security: system test definition and mock definition.

There are some limitation to this, but we find that it's possible to reliably cover ⪆95% of APIs using this approach.