Strimzi
-
|
Hello. I deploy Kafka Connect using Strimzi with the same configuration across more than two K8S clusters. However, when the Kafka Connect cluster is configured in this way, workers cannot find the leader worker, making the REST API unusable. I suspect this is because the rest.advertised.host.name is set to the worker's name, which prevents communication to workers in other K8S clusters. If, by chance, API requests reach the K8S cluster where the leader worker is located, the API functions properly. However, even in this situation, communication between workers fails, preventing task creation. Therefore, I thought that changing the rest.advertised.host.name to the host IP and exposing it via nodeport might solve the issue of enabling communication between worker nodes located in different K8S clusters. QuestionDoes Strimzi not support using multiple K8S clusters as a single Kafka Connect cluster in this manner? My ConfigBelow is my Strimzi K8S config. It is being deployed identically on two K8S clusters. (excluding the ingress-related configurations)
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
name: src-ct
labels:
k8s-app: src-ct
app: src-ct
spec:
version: 3.8.0
replicas: {{ .Values.kafkaConnect.replicas }}
bootstrapServers: {{ .Values.kafkaConnect.bootstrapServers }}
image: {{ .Values.kafkaConnect.image }}
config:
group.id: {{ .Values.kafkaConnect.groupId }}
offset.storage.topic: {{ .Values.kafkaConnect.storageTopic }}-offsets
config.storage.topic: {{ .Values.kafkaConnect.storageTopic }}-configs
status.storage.topic: {{ .Values.kafkaConnect.storageTopic }}-statuses
config.storage.replication.factor: -1
offset.storage.replication.factor: -1
status.storage.replication.factor: -1
key.converter: io.confluent.connect.avro.AvroConverter
key.converter.schema.registry.url: {{ .Values.schemaRegistry}}
value.converter: io.confluent.connect.avro.AvroConverter
value.converter.schema.registry.url: {{ .Values.schemaRegistry}}
producer.max.request.size: 10485760 # 10MB
logging:
type: inline
loggers:
log4j.rootLogger: {{ .Values.kafkaConnect.logging.rootLogger }}
log4j.logger.io.debezium.connector.jdbc: {{ .Values.kafkaConnect.logging.jdbcLogger }}
log4j.logger.io.debezium.connector.mysql: {{ .Values.kafkaConnect.logging.mysqlLogger }}
log4j.logger.org.apache.kafka.connect.runtime.rest.RestServer: {{ .Values.kafkaConnect.logging.restServerLogger }}
jvmOptions:
javaSystemProperties:
- name: file.encoding
value: "UTF-8"
- name: networkaddress.cache.ttl
value: "0"
- name: networkaddress.cache.negative.ttl
value: "0"
template:
pod:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 10
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- src-ct
topologyKey: "kubernetes.io/hostname"
---
apiVersion: v1
kind: Service
metadata:
name: src-ct-lb
labels:
app: kafka-connect
spec:
type: LoadBalancer
ports:
- name: external
port: 8083
targetPort: 8083
protocol: TCP
selector:
strimzi.io/cluster: src-ct
strimzi.io/kind: KafkaConnect
logThanks. |
Beta Was this translation helpful? Give feedback.
Answered by
scholzj
Feb 6, 2025
Replies: 1 comment
-
|
That is not supported and will not work currently. |
Beta Was this translation helpful? Give feedback.
0 replies
Answer selected by
jooomOut
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
That is not supported and will not work currently.