Same issues as #2218 - consumergroup gets stuck and cannot restart

[richardvflux]

Describe the bug
It seems when a timeout (client-disconnect, not sure if the error matters) occurs then the consumergroup goes into a tailspin and can never reconnect because the group is endlessly rebalancing. We have no setting for the group rebalance delay so its default at 3s. All our other kafka settings are default values.

Expected behavior
It should recover and continue processing.

To Reproduce
We are not sure what is causing it so we cannot reproduce it, its seems random.

Knative release version
Eventing is 1.14.14

Additional context
We don't have a workaround, we are having to play whackamole with the logs.

{"@timestamp":"2024-12-10T21:07:26.562Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-statement-manager-request-priority-trigger-19, groupId=knative-trigger-bbt-1310-statement-manager-request-priority-trigger] Request joining group due to: group is already rebalancing","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"kafka-coordinator-heartbeat-thread | knative-trigger-bbt-1310-statement-manager-request-priority-trigger","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-10T21:07:27.099Z","@version":"1","message":"failed to send event to subscriber context={topics=[knative-broker-bbt-1310-default], consumerGroup='knative-trigger-bbt-1310-statement-manager-request-priority-trigger', reference=uuid: \"5a929a33-c719-4dc0-aac1-d9464593ea0b\"\nnamespace: \"bbt-1310\"\nname: \"statement-manager-request-priority-trigger\"\n} target=http://statement-manager-app.bbt-1310.svc.cluster.local/event","logger_name":"dev.knative.eventing.kafka.broker.dispatcher.impl.http.WebClientCloudEventSender","thread_name":"vert.x-eventloop-thread-2","level":"ERROR","level_value":40000,"stack_trace":"java.lang.IllegalStateException: Client is closed\n\tat io.vertx.core.http.impl.HttpClientImpl.checkClosed(HttpClientImpl.java:405)\n\tat io.vertx.core.http.impl.HttpClientImpl.doRequest(HttpClientImpl.java:281)\n\tat io.vertx.core.http.impl.HttpClientImpl.request(HttpClientImpl.java:191)\n\tat io.vertx.ext.web.client.impl.HttpContext.handleCreateRequest(HttpContext.java:488)\n\tat io.vertx.ext.web.client.impl.HttpContext.execute(HttpContext.java:372)\n\tat io.vertx.ext.web.client.impl.HttpContext.next(HttpContext.java:362)\n\tat io.vertx.ext.web.client.impl.HttpContext.fire(HttpContext.java:329)\n\tat io.vertx.ext.web.client.impl.HttpContext.createRequest(HttpContext.java:220)\n\tat io.vertx.ext.web.client.impl.HttpContext.handlePrepareRequest(HttpContext.java:418)\n\tat io.vertx.ext.web.client.impl.HttpContext.execute(HttpContext.java:369)\n\tat io.vertx.ext.web.client.impl.HttpContext.next(HttpContext.java:362)\n\tat io.vertx.ext.web.client.impl.HttpContext.fire(HttpContext.java:329)\n\tat io.vertx.ext.web.client.impl.HttpContext.prepareRequest(HttpContext.java:208)\n\tat io.vertx.ext.web.client.impl.HttpRequestImpl.send(HttpRequestImpl.java:538)\n\tat io.vertx.ext.web.client.impl.HttpRequestImpl.sendBuffer(HttpRequestImpl.java:425)\n\tat io.vertx.ext.web.client.HttpRequest.sendBuffer(HttpRequest.java:533)\n\tat io.cloudevents.http.vertx.impl.VertxWebClientRequestMessageWriterImpl.end(VertxWebClientRequestMessageWriterImpl.java:62)\n\tat io.cloudevents.http.vertx.impl.VertxWebClientRequestMessageWriterImpl.end(VertxWebClientRequestMessageWriterImpl.java:32)\n\tat io.cloudevents.core.impl.BaseCloudEvent.read(BaseCloudEvent.java:61)\n\tat io.cloudevents.rw.CloudEventReader.read(CloudEventReader.java:43)\n\tat io.cloudevents.core.message.MessageWriter.writeBinary(MessageWriter.java:71)\n\tat dev.knative.eventing.kafka.broker.dispatcher.impl.http.WebClientCloudEventSender.lambda$send$4(WebClientCloudEventSender.java:223)\n\tat io.vertx.core.impl.future.Composition.onSuccess(Composition.java:38)\n\tat io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:66)\n\tat io.vertx.core.impl.future.SucceededFuture.addListener(SucceededFuture.java:88)\n\tat io.vertx.core.impl.future.FutureBase.compose(FutureBase.java:90)\n\tat io.vertx.core.impl.future.SucceededFuture.compose(SucceededFuture.java:27)\n\tat io.vertx.core.Future.compose(Future.java:375)\n\tat dev.knative.eventing.kafka.broker.dispatcher.impl.http.WebClientCloudEventSender.send(WebClientCloudEventSender.java:200)\n\tat dev.knative.eventing.kafka.broker.dispatcher.impl.http.WebClientCloudEventSender.send(WebClientCloudEventSender.java:130)\n\tat dev.knative.eventing.kafka.broker.dispatcher.impl.http.WebClientCloudEventSender.lambda$retry$3(WebClientCloudEventSender.java:176)\n\tat io.vertx.core.impl.VertxImpl$InternalTimerHandler.handle(VertxImpl.java:1056)\n\tat io.vertx.core.impl.VertxImpl$InternalTimerHandler.handle(VertxImpl.java:1027)\n\tat io.vertx.core.impl.ContextImpl.emit(ContextImpl.java:328)\n\tat io.vertx.core.impl.ContextImpl.emit(ContextImpl.java:321)\n\tat io.vertx.core.impl.ContextInternal.emit(ContextInternal.java:209)\n\tat io.vertx.core.impl.VertxImpl$InternalTimerHandler.run(VertxImpl.java:1045)\n\tat io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)\n\tat io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:153)\n\tat io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)\n\tat io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Unknown Source)\n","context":{},"target":"http://statement-manager-app.bbt-1310.svc.cluster.local/event"}
{"@timestamp":"2024-12-10T21:07:29.565Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-statement-manager-request-priority-trigger-19, groupId=knative-trigger-bbt-1310-statement-manager-request-priority-trigger] Request joining group due to: group is already rebalancing","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"kafka-coordinator-heartbeat-thread | knative-trigger-bbt-1310-statement-manager-request-priority-trigger","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-10T21:07:32.567Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-statement-manager-request-priority-trigger-19, groupId=knative-trigger-bbt-1310-statement-manager-request-priority-trigger] Request joining group due to: group is already rebalancing","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"kafka-coordinator-heartbeat-thread | knative-trigger-bbt-1310-statement-manager-request-priority-trigger","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-10T21:07:35.57Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-statement-manager-request-priority-trigger-19, groupId=knative-trigger-bbt-1310-statement-manager-request-priority-trigger] Request joining group due to: group is already rebalancing","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"kafka-coordinator-heartbeat-thread | knative-trigger-bbt-1310-statement-manager-request-priority-trigger","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-10T21:07:38.572Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-statement-manager-request-priority-trigger-19, groupId=knative-trigger-bbt-1310-statement-manager-request-priority-trigger] Request joining group due to: group is already rebalancing","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"kafka-coordinator-heartbeat-thread | knative-trigger-bbt-1310-statement-manager-request-priority-trigger","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-10T21:07:41.575Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-statement-manager-request-priority-trigger-19, groupId=knative-trigger-bbt-1310-statement-manager-request-priority-trigger] Request joining group due to: group is already rebalancing","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"kafka-coordinator-heartbeat-thread | knative-trigger-bbt-1310-statement-manager-request-priority-trigger","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-10T21:07:44.578Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-statement-manager-request-priority-trigger-19, groupId=knative-trigger-bbt-1310-statement-manager-request-priority-trigger] Request joining group due to: group is already rebalancing","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"kafka-coordinator-heartbeat-thread | knative-trigger-bbt-1310-statement-manager-request-priority-trigger","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-10T21:07:47.58Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-statement-manager-request-priority-trigger-19, groupId=knative-trigger-bbt-1310-statement-manager-request-priority-trigger] Request joining group due to: group is already rebalancing","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"kafka-coordinator-heartbeat-thread | knative-trigger-bbt-1310-statement-manager-request-priority-trigger","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-10T21:07:50.583Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-statement-manager-request-priority-trigger-19, groupId=knative-trigger-bbt-1310-statement-manager-request-priority-trigger] Request joining group due to: group is already rebalancing","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"kafka-coordinator-heartbeat-thread | knative-trigger-bbt-1310-statement-manager-request-priority-trigger","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-10T21:07:53.585Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-statement-manager-request-priority-trigger-19, groupId=knative-trigger-bbt-1310-statement-manager-request-priority-trigger] Request joining group due to: group is already rebalancing","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"kafka-coordinator-heartbeat-thread | knative-trigger-bbt-1310-statement-manager-request-priority-trigger","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-10T21:07:56.588Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-statement-manager-request-priority-trigger-19, groupId=knative-trigger-bbt-1310-statement-manager-request-priority-trigger] Request joining group due to: group is already rebalancing","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"kafka-coordinator-heartbeat-thread | knative-trigger-bbt-1310-statement-manager-request-priority-trigger","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-10T21:07:59.591Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-statement-manager-request-priority-trigger-19, groupId=knative-trigger-bbt-1310-statement-manager-request-priority-trigger] Request joining group due to: group is already rebalancing","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"kafka-coordinator-heartbeat-thread | knative-trigger-bbt-1310-statement-manager-request-priority-trigger","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-10T21:08:02.593Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-statement-manager-request-priority-trigger-19, groupId=knative-trigger-bbt-1310-statement-manager-request-priority-trigger] Request joining group due to: group is already rebalancing","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"kafka-coordinator-heartbeat-thread | knative-trigger-bbt-1310-statement-manager-request-priority-trigger","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-10T21:08:05.596Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-statement-manager-request-priority-trigger-19, groupId=knative-trigger-bbt-1310-statement-manager-request-priority-trigger] Request joining group due to: group is already rebalancing","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"kafka-coordinator-heartbeat-thread | knative-trigger-bbt-1310-statement-manager-request-priority-trigger","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-10T21:08:08.598Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-statement-manager-request-priority-trigger-19, groupId=knative-trigger-bbt-1310-statement-manager-request-priority-trigger] Request joining group due to: group is already rebalancing","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"kafka-coordinator-heartbeat-thread | knative-trigger-bbt-1310-statement-manager-request-priority-trigger","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-10T21:08:11.601Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-statement-manager-request-priority-trigger-19, groupId=knative-trigger-bbt-1310-statement-manager-request-priority-trigger] Request joining group due to: group is already rebalancing","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"kafka-coordinator-heartbeat-thread | knative-trigger-bbt-1310-statement-manager-request-priority-trigger","level":"INFO","level_value":20000}
[etc]
{"@timestamp":"2024-12-10T21:12:19.09Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-statement-manager-request-priority-trigger-44, groupId=knative-trigger-bbt-1310-statement-manager-request-priority-trigger] JoinGroup failed: The coordinator is not aware of this member. Need to re-join the group. Sent generation was Generation{generationId=-1, memberId='consumer-knative-trigger-bbt-1310-statement-manager-request-priority-trigger-44-858f0aec-508d-4048-baa1-e729800d29bb', protocol='null'}","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"Thread-223","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-10T21:12:19.09Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-statement-manager-request-priority-trigger-44, groupId=knative-trigger-bbt-1310-statement-manager-request-priority-trigger] Resetting generation and member id due to: encountered UNKNOWN_MEMBER_ID from JOIN_GROUP response","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"Thread-223","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-10T21:12:19.09Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-statement-manager-request-priority-trigger-44, groupId=knative-trigger-bbt-1310-statement-manager-request-priority-trigger] Request joining group due to: encountered UNKNOWN_MEMBER_ID from JOIN_GROUP response","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"Thread-223","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-10T21:12:19.09Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-statement-manager-request-priority-trigger-44, groupId=knative-trigger-bbt-1310-statement-manager-request-priority-trigger] Request joining group due to: rebalance failed due to 'The coordinator is not aware of this member.' (UnknownMemberIdException)","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"Thread-223","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-10T21:12:19.09Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-statement-manager-request-priority-trigger-44, groupId=knative-trigger-bbt-1310-statement-manager-request-priority-trigger] (Re-)joining group","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"Thread-223","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-10T21:12:19.092Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-statement-manager-request-priority-trigger-44, groupId=knative-trigger-bbt-1310-statement-manager-request-priority-trigger] Request joining group due to: need to re-join with the given member-id: consumer-knative-trigger-bbt-1310-statement-manager-request-priority-trigger-44-7695e733-2f01-41d9-a0dd-6051435833f5","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"Thread-223","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-10T21:12:19.092Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-statement-manager-request-priority-trigger-44, groupId=knative-trigger-bbt-1310-statement-manager-request-priority-trigger] Request joining group due to: rebalance failed due to 'The group member needs to have a valid member id before actually entering a consumer group.' (MemberIdRequiredException)","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"Thread-223","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-10T21:12:19.092Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-statement-manager-request-priority-trigger-44, groupId=knative-trigger-bbt-1310-statement-manager-request-priority-trigger] (Re-)joining group","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"Thread-223","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-10T21:12:20.816Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-statement-manager-request-priority-trigger-19, groupId=knative-trigger-bbt-1310-statement-manager-request-priority-trigger] Request joining group due to: group is already rebalancing","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"kafka-coordinator-heartbeat-thread | knative-trigger-bbt-1310-statement-manager-request-priority-trigger","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-10T21:12:23.819Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-statement-manager-request-priority-trigger-19, groupId=knative-trigger-bbt-1310-statement-manager-request-priority-trigger] Request joining group due to: group is already rebalancing","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"kafka-coordinator-heartbeat-thread | knative-trigger-bbt-1310-statement-manager-request-priority-trigger","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-10T21:12:26.822Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-statement-manager-request-priority-trigger-19, groupId=knative-trigger-bbt-1310-statement-manager-request-priority-trigger] Request joining group due to: group is already rebalancing","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"kafka-coordinator-heartbeat-thread | knative-trigger-bbt-1310-statement-manager-request-priority-trigger","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-10T21:12:29.824Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-statement-manager-request-priority-trigger-19, groupId=knative-trigger-bbt-1310-statement-manager-request-priority-trigger] Request joining group due to: group is already rebalancing","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"kafka-coordinator-heartbeat-thread | knative-trigger-bbt-1310-statement-manager-request-priority-trigger","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-10T21:12:32.827Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-statement-manager-request-priority-trigger-19, groupId=knative-trigger-bbt-1310-statement-manager-request-priority-trigger] Request joining group due to: group
[etc]

[richardvflux]

more logs after running on debug for a day

{"@timestamp":"2024-12-12T02:03:50.784Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-internal-engine-cloud-events-trigger-244, groupId=knative-trigger-bbt-1310-internal-engine-cloud-events-trigger] Subscribed to topic(s): knative-broker-bbt-1310-default","logger_name":"org.apache.kafka.clients.consumer.KafkaConsumer","thread_name":"Thread-1223","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-12T02:03:50.802Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-internal-engine-cloud-events-trigger-244, groupId=knative-trigger-bbt-1310-internal-engine-cloud-events-trigger] Resetting the last seen epoch of partition knative-broker-bbt-1310-default-0 to 0 since the associated topicId changed from null to NUqfRFk-RbqABGv27xB8tA","logger_name":"org.apache.kafka.clients.Metadata","thread_name":"Thread-1223","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-12T02:03:50.802Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-internal-engine-cloud-events-trigger-244, groupId=knative-trigger-bbt-1310-internal-engine-cloud-events-trigger] Resetting the last seen epoch of partition knative-broker-bbt-1310-default-2 to 0 since the associated topicId changed from null to NUqfRFk-RbqABGv27xB8tA","logger_name":"org.apache.kafka.clients.Metadata","thread_name":"Thread-1223","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-12T02:03:50.802Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-internal-engine-cloud-events-trigger-244, groupId=knative-trigger-bbt-1310-internal-engine-cloud-events-trigger] Resetting the last seen epoch of partition knative-broker-bbt-1310-default-1 to 0 since the associated topicId changed from null to NUqfRFk-RbqABGv27xB8tA","logger_name":"org.apache.kafka.clients.Metadata","thread_name":"Thread-1223","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-12T02:03:50.802Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-internal-engine-cloud-events-trigger-244, groupId=knative-trigger-bbt-1310-internal-engine-cloud-events-trigger] Cluster ID: Z1mVpsiZSTChR86eXkqF-A","logger_name":"org.apache.kafka.clients.Metadata","thread_name":"Thread-1223","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-12T02:03:50.802Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-internal-engine-cloud-events-trigger-244, groupId=knative-trigger-bbt-1310-internal-engine-cloud-events-trigger] Discovered group coordinator b-1.psauqamsk.wekpd9.c3.kafka.ap-southeast-2.amazonaws.com:9096 (id: 2147483646 rack: null)","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"Thread-1223","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-12T02:03:50.803Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-internal-engine-cloud-events-trigger-244, groupId=knative-trigger-bbt-1310-internal-engine-cloud-events-trigger] (Re-)joining group","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"Thread-1223","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-12T02:03:50.803Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-internal-engine-cloud-events-trigger-230, groupId=knative-trigger-bbt-1310-internal-engine-cloud-events-trigger] Member consumer-knative-trigger-bbt-1310-internal-engine-cloud-events-trigger-230-1e23bc03-68a0-4f74-899f-befa843c427d sending LeaveGroup request to coordinator b-1.psauqamsk.wekpd9.c3.kafka.ap-southeast-2.amazonaws.com:9096 (id: 2147483646 rack: null) due to the consumer is being closed","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"Thread-1153","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-12T02:03:50.803Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-internal-engine-cloud-events-trigger-230, groupId=knative-trigger-bbt-1310-internal-engine-cloud-events-trigger] Resetting generation and member id due to: consumer pro-actively leaving the group","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"Thread-1153","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-12T02:03:50.803Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-internal-engine-cloud-events-trigger-230, groupId=knative-trigger-bbt-1310-internal-engine-cloud-events-trigger] Request joining group due to: consumer pro-actively leaving the group","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"Thread-1153","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-12T02:03:50.819Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-internal-engine-cloud-events-trigger-244, groupId=knative-trigger-bbt-1310-internal-engine-cloud-events-trigger] Request joining group due to: need to re-join with the given member-id: consumer-knative-trigger-bbt-1310-internal-engine-cloud-events-trigger-244-0928acb1-1f82-40d5-b701-119bb78fcd8a","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"Thread-1223","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-12T02:03:50.819Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-internal-engine-cloud-events-trigger-244, groupId=knative-trigger-bbt-1310-internal-engine-cloud-events-trigger] Request joining group due to: rebalance failed due to 'The group member needs to have a valid member id before actually entering a consumer group.' (MemberIdRequiredException)","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"Thread-1223","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-12T02:03:50.819Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-internal-engine-cloud-events-trigger-244, groupId=knative-trigger-bbt-1310-internal-engine-cloud-events-trigger] (Re-)joining group","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"Thread-1223","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-12T02:03:52.771Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-internal-engine-cloud-events-trigger-230, groupId=knative-trigger-bbt-1310-internal-engine-cloud-events-trigger] Failed to close coordinator","logger_name":"org.apache.kafka.clients.consumer.KafkaConsumer","thread_name":"Thread-1153","level":"ERROR","level_value":40000,"stack_trace":"org.apache.kafka.common.errors.InterruptException: java.lang.InterruptedException\n\tat org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.maybeThrowInterruptException(ConsumerNetworkClient.java:535)\n\tat org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:296)\n\tat org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:251)\n\tat org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:242)\n\tat org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitPendingRequests(ConsumerNetworkClient.java:353)\n\tat org.apache.kafka.clients.consumer.internals.AbstractCoordinator.close(AbstractCoordinator.java:1100)\n\tat org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.close(ConsumerCoordinator.java:1048)\n\tat org.apache.kafka.clients.consumer.KafkaConsumer.close(KafkaConsumer.java:2411)\n\tat org.apache.kafka.clients.consumer.KafkaConsumer.close(KafkaConsumer.java:2378)\n\tat org.apache.kafka.clients.consumer.KafkaConsumer.close(KafkaConsumer.java:2351)\n\tat dev.knative.eventing.kafka.broker.dispatcherloom.LoomKafkaConsumer.lambda$close$1(LoomKafkaConsumer.java:109)\n\tat dev.knative.eventing.kafka.broker.dispatcherloom.LoomKafkaConsumer.processTaskQueue(LoomKafkaConsumer.java:75)\n\tat java.base/java.lang.Thread.run(Unknown Source)\nCaused by: java.lang.InterruptedException: null\n\t... 13 common frames omitted\n"}
{"@timestamp":"2024-12-12T02:03:52.772Z","@version":"1","message":"App info kafka.consumer for consumer-knative-trigger-bbt-1310-internal-engine-cloud-events-trigger-230 unregistered","logger_name":"org.apache.kafka.common.utils.AppInfoParser","thread_name":"Thread-1153","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-12T02:03:52.772Z","@version":"1","message":"Consumer verticle closed context={topics=[knative-broker-bbt-1310-default], consumerGroup='knative-trigger-bbt-1310-internal-engine-cloud-events-trigger', reference=uuid: \"23d712ce-5eaf-41a7-a5b1-e59190291e9c\"\nnamespace: \"bbt-1310\"\nname: \"internal-engine-cloud-events-trigger\"\n}","logger_name":"dev.knative.eventing.kafka.broker.dispatcher.impl.consumer.ConsumerVerticle","thread_name":"Thread-1153","level":"INFO","level_value":20000,"context":{}}
{"@timestamp":"2024-12-12T02:03:53.04Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-internal-engine-cloud-events-trigger-73, groupId=knative-trigger-bbt-1310-internal-engine-cloud-events-trigger] Request joining group due to: group is already rebalancing","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"kafka-coordinator-heartbeat-thread | knative-trigger-bbt-1310-internal-engine-cloud-events-trigger","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-12T02:03:56.043Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-internal-engine-cloud-events-trigger-73, groupId=knative-trigger-bbt-1310-internal-engine-cloud-events-trigger] Request joining group due to: group is already rebalancing","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"kafka-coordinator-heartbeat-thread | knative-trigger-bbt-1310-internal-engine-cloud-events-trigger","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-12T02:03:59.046Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-internal-engine-cloud-events-trigger-73, groupId=knative-trigger-bbt-1310-internal-engine-cloud-events-trigger] Request joining group due to: group is already rebalancing","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"kafka-coordinator-heartbeat-thread | knative-trigger-bbt-1310-internal-engine-cloud-events-trigger","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-12T02:04:02.048Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-internal-engine-cloud-events-trigger-73, groupId=knative-trigger-bbt-1310-internal-engine-cloud-events-trigger] Request joining group due to: group is already rebalancing","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"kafka-coordinator-heartbeat-thread | knative-trigger-bbt-1310-internal-engine-cloud-events-trigger","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-12T02:04:05.051Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-internal-engine-cloud-events-trigger-73, groupId=knative-trigger-bbt-1310-internal-engine-cloud-events-trigger] Request joining group due to: group is already rebalancing","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"kafka-coordinator-heartbeat-thread | knative-trigger-bbt-1310-internal-engine-cloud-events-trigger","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-12T02:04:08.054Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-internal-engine-cloud-events-trigger-73, groupId=knative-trigger-bbt-1310-internal-engine-cloud-events-trigger] Request joining group due to: group is already rebalancing","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"kafka-coordinator-heartbeat-thread | knative-trigger-bbt-1310-internal-engine-cloud-events-trigger","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-12T02:04:11.057Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-internal-engine-cloud-events-trigger-73, groupId=knative-trigger-bbt-1310-internal-engine-cloud-events-trigger] Request joining group due to: group is already rebalancing","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"kafka-coordinator-heartbeat-thread | knative-trigger-bbt-1310-internal-engine-cloud-events-trigger","level":"INFO","level_value":20000}
{"@timestamp":"2024-12-12T02:04:14.059Z","@version":"1","message":"[Consumer clientId=consumer-knative-trigger-bbt-1310-internal-engine-cloud-events-trigger-73, groupId=knative-trigger-bbt-1310-internal-engine-cloud-events-trigger] Request joining group due to: group is already rebalancing","logger_name":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","thread_name":"kafka-coordinator-heartbeat-thread | knative-trigger-bbt-1310-internal-engine-cloud-events-trigger","level":"INFO","level_value":20000}