[exporter/elasticsearch]: add automatic mapping mode

[mauri870]

Description

This adds support for detecting the mapping mode based on a metadata field set by the client, this mapping mode takes precedence over the configuration.

Link to tracking issue

Fixes #36092

Testing

Unit tests added for the presence/absence of the header value.

Documentation

Added README docs next to existing mapping mode section.

[carsonip]

TODO to self: check if RequireDataStream still works, as it mostly won't. OTel mode now needs RequireDataStream at the top level during bulk indexer initialization to avoid race conditions where ES otel data plugin isn't ready, resulting in legacy index being created / ingest failures. With automatic mapping mode, we may need require_data_stream to be enabled or disabled per-request, which may not be possible at the moment.

[carsonip]

What you implemented here is to always check for the header and override the mapping mode config. However, my understanding of the internal spec is that we should expose it as a config value "auto", and when it is set, it will switch mapping mode per request based on header.

[mauri870]

Thanks for bringing that up; I misunderstood the specs at first. So, the auto-mapping mode should only kick in if it was explicitly allowed in the config. I'll make the changes accordingly.

[mauri870]

OTel mode now needs RequireDataStream at the top level during bulk indexer initialization ... may need require_data_stream to be enabled or disabled per-request, which may not be possible at the moment.

I have noticed that as well, and agree that there is not an easy way to handle this at the moment. I can't think of a way to circunvent this currently without a major refactoring.

[carsonip]

I have noticed that as well, and agree that there is not an easy way to handle this at the moment. I can't think of a way to circunvent this currently without a major refactoring.

We can either

• make require_data_stream per-request, but that means we'll have to bump elasticsearch client and expose that in docappender as well, which may involve a few blockers; or
• if the use case of automatic mapping mode expects index templates for data stream to be present, then let's call it a limitation of automatic mapping mode. In that case, we can set require_data_stream for both "otel" and "auto" mapping mode during initialization.

[felixbarny]

Sounds like a reasonable limitation that the auto mode requires data streams. Ingesting into plain indices seems like an edge case.

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 14 days.