You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When under a huge load of incoming messages, the consumer crashes with errors "commitOffset is not a function" or "fetch is not a function".
I updated the configuration to
Hello, our software is facing some issues while reading data from Kafka ("kafkajs": "^2.2.4").
The consumer is reading messages from a topic with two partitions. Below is the initial consumer configuration:
.consumer.run({ autoCommit: true, autocommitThreshold: 1000, eachMessage: async ({ topic, partition, message }) => { if (message.headers?.channel) { // manage message } else { // manage message } } });
When under a huge load of incoming messages, the consumer crashes with errors "commitOffset is not a function" or "fetch is not a function".
I updated the configuration to
consumer.run({ autoCommit: true, eachMessage: async ({ topic, partition, message }) => { if (message.headers?.channel) { // manage message } else { // manage message } } });
so that no
autocommitThreshold
was specified. Same errors "commitOffset is not a function" or "fetch is not a function".To avoid the issue, the following configuration has been added:
autocommitThreshold: 1
The problem is that the workaround is slowing down the process since the consumer is forced to commit the offset after every message.
Is there any additional configuration to be added to avoid the commit after 1 message and to speed up the process?
Thanks.
The text was updated successfully, but these errors were encountered: