[Bug]: jaeger-es-rollover-init tries to create already existing [jaeger-span-000001] index in Elasticseach

[sergeykad]

What happened?

I am redeploying Jaeger Helm chart on a Kubernetes namespace. During the deployment, jaeger-es-rollover-init tries to create the already existing index and fails, failing the whole deployment. I am not sure what caused it since I redeployed it a few times successfully before.

The only change I can think of is that the Spark job was recently added to the deployment.

I am using the following configuration.

esRollover:
  enabled: true
  cmdlineParams:
    es.use-ilm: "true"
    es.ilm-policy-name: "30-days-default"
  initHook:
    extraEnv:
      - name: SHARDS
        value: "3"
      - name: REPLICAS
        value: "0"

Steps to reproduce

1. Deploy Jaeger
2. Add Spark job
3. Redeploy

Spark configuration

spark:
  enabled: true
  schedule: "30 7 * * *"
  image:
    registry: ghcr.io
    repository: jaegertracing/spark-dependencies/spark-dependencies
  extraEnv:
    - name: ES_USE_ALIASES
      value: "true"
    - name: JAVA_OPTS
      value: "-Xmx64G"

Expected behavior

Jaeger deployed succesfully

Relevant log output

Error: failed to create index: jaeger-span-000001, request failed, status code: 400, body: {"error":{"root_cause":[{"type":"invalid_index_name_exception","reason":"Invalid index name [jaeger-span-000001], already exists as alias","index_uuid":"_na_","index":"jaeger-span-000001"}],"type":"invalid_index_name_exception","reason":"Invalid index name [jaeger-span-000001], already exists as alias","index_uuid":"_na_","index":"jaeger-span-000001"},"status":400}

Screenshot

No response

Additional context

Jaeger 3.2.0
Kubernetes 1.25.6
Elasticsearch 8.9.0

Jaeger backend version

3.2.0

SDK

OpenTelemetry Operator

Pipeline

OTEL SDK -> Jaeger Collector -> Elasticsearch

Stogage backend

Elasticsearch 8.9.0

Operating system

No response

Deployment model

Kubernetes

Deployment configs

[Manik2708]

This is weird, init was made indempotent, I tested running init multple times but never got this error, this needs investigation!

[sergeykad]

This happens consistently on our servers. We had to disable the init configuration to work around this.

[Manik2708]

Did it use to happen before addition of spark also?

[sergeykad]

I am not sure about it.