[Bug]: KafkaUser object doesn't produce a new certificate after the kafka cluster is destroyed and redeployed

[varunreddyj]

Bug Description

once the strimzi kafka operator is deployed, we are deployed kafka cluster, we are deplying KafkaUser from our application manifest which in turn will create secrets with certs. But when we delete the kafka cluster and redeploy it , KafkaUser object doesn't produce a new certificate and hence causing SSL Handshake issues when our application is trying to communicate with kafka cluster .

Steps to reproduce

1. Deploy strimzi operator
2. Deploy Kafka cluster
3. create KafkaUser object which will create secrets with certs to communicate with kafka cluster
4. delete kafka cluster
5. Re-deploy kafka cluster
6. existing kafkauser wont be able to communicate with new kafka because KafkaUser object doesn't produce a new certificate after the kafka cluster is destroyed and redeployed.

Expected behavior

KafkaUser object should produce a new certificate after the kafka cluster is destroyed and redeployed

Strimzi version

0.42.0

Kubernetes version

v1.31.1+rke2r1.stable.0

Installation method

Helm Chart

Infrastructure

RKE2 on EC2 instances

Configuration files and logs

2024-11-04 18:09:13,036 INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication with /10.42.0.240 (channelId=10.42.3.249:9093-10.42.0.240:56800-35902) (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-threa │
│ d-0-ListenerName(TLS-9093)-SSL-9] │
│ 2024-11-04 18:09:13,116 INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication with /10.42.0.38 (channelId=10.42.3.249:9093-10.42.0.38:52330-35903) (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread- │
│ 0-ListenerName(TLS-9093)-SSL-7] │
│ 2024-11-04 18:09:13,146 INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication with /10.42.2.180 (channelId=10.42.3.249:9093-10.42.2.180:33556-35903) (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-threa │
│ d-0-ListenerName(TLS-9093)-SSL-8] │
│ 2024-11-04 18:09:13,156 INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication with /10.42.1.109 (channelId=10.42.3.249:9093-10.42.1.109:47292-35903) (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-threa │
│ d-0-ListenerName(TLS-9093)-SSL-9] │
│ 2024-11-04 18:09:13,209 INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication with /10.42.2.192 (channelId=10.42.3.249:9093-10.42.2.192:47260-35904) (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-threa │
│ d-0-ListenerName(TLS-9093)-SSL-7] │
│ 2024-11-04 18:09:13,264 INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication with /10.42.2.212 (channelId=10.42.3.249:9093-10.42.2.212:34028-35904) (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-threa │
│ d-0-ListenerName(TLS-9093)-SSL-8] │
│ 2024-11-04 18:09:13,270 INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication with /10.42.2.209 (channelId=10.42.3.249:9093-10.42.2.209:42968-35905) (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-threa │
│ d-0-ListenerName(TLS-9093)-SSL-7] │
│ 2024-11-04 18:09:13,278 INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication with /10.42.2.172 (channelId=10.42.3.249:9093-10.42.2.172:53968-35905) (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-threa │
│ d-0-ListenerName(TLS-9093)-SSL-8] │
│ 2024-11-04 18:09:13,324 INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication with /10.42.0.248 (channelId=10.42.3.249:9093-10.42.0.248:43318-35905) (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-threa │
│ d-0-ListenerName(TLS-9093)-SSL-9] │
│ 2024-11-04 18:09:13,364 INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication with /10.42.0.34 (channelId=10.42.3.249:9093-10.42.0.34:48808-35904) (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread- │
│ 0-ListenerName(TLS-9093)-SSL-9]

Additional context

No response

[im-konge]

Hi, could you please attach logs from the User Operator? Also, in what state is the KafkaUser CR?

[scholzj]

I think this needs to have all custom resources provided and the full logs from all components as well. Not just from user operator and the Kafka CR.

[im-konge]

Triaged on 14.11.2024: @varunreddyj could you please attach the logs and YAMLs of the CRs, so we can have a look at it? Thanks

[scholzj]

FYI: I tries to reproduce this, but without success. So unless more information is provided, we should close it.