[BUG] - k8s integration ingestion method is not clear enough

[shaigbdb]

What is the bug?
We've been able to install the integration dashboards as well as the sample data. However, it is unclear how one is expected to ingest the relevant data.

The repo mentions using OpenTelemetry collectors to collect the data. However, the documentation in the OpenTelemetry website is insufficient for this - I'm getting errors from the opentelemetry pod.

2024-12-12T13:35:10.976Z	error	internal/queue_sender.go:92	Exporting failed. Dropping data.	{"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "not retryable error: Permanent error: rpc error: code = Unimplemented desc = unexpected HTTP status code received from server: 404 (Not Found); transport: received unexpected content-type \"text/plain; charset=utf-8\"", "dropped_items": 1}
go.opentelemetry.io/collector/exporter/exporterhelper/internal.NewQueueSender.func1
	go.opentelemetry.io/collector/[email protected]/exporterhelper/internal/queue_sender.go:92
go.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1
	go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:43

I've also tried the fluentbit configuration though unfortunately it seems to be irrelevant. The fluentbit kubernetes filter only collects partial data from Kubernetes which does not match the data required by the integration's dashboards (such as the deployment details).

There is a fluent bit issue describing this:

fluent/fluent-bit#9263

The filebeat versions supported by Opensearch have a similar issue, so it is not an option. The other options mentioned in that issue require developing support for the additional fields.

How can one reproduce the bug?
Use attached data prepper and otel configurations in accordance with the documentation.

What is the expected behavior?
Kubernetes integration data being loaded into the cluster.
I would appreciate templates for data prepper and otel (assuming that is the solution) that correctly collect the integration data for k8s.

What is your host/environment?

• OS: Linux (EKS)
• Opensearch 2.14
  data-prepper-k8s.yaml.txt
  values.yaml.txt

Thanks.

[YANG-DB]

@shaigbdb - thanks for this feedback !
we will review and propose some documentation on better End2End and possibly (a bit later) a k8s based solution showcasing the entire workflow

[krisfreedain]

Catch All Triage - 1, 2, 3