Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

consumer error CommitOffset is not a function #1715

Open
gnpace opened this issue Sep 16, 2024 · 1 comment
Open

consumer error CommitOffset is not a function #1715

gnpace opened this issue Sep 16, 2024 · 1 comment

Comments

@gnpace
Copy link

gnpace commented Sep 16, 2024

Hello, our software is facing some issues while reading data from Kafka ("kafkajs": "^2.2.4").

The consumer is reading messages from a topic with two partitions. Below is the initial consumer configuration:

.consumer.run({ autoCommit: true, autocommitThreshold: 1000, eachMessage: async ({ topic, partition, message }) => { if (message.headers?.channel) { // manage message } else { // manage message } } });

When under a huge load of incoming messages, the consumer crashes with errors "commitOffset is not a function" or "fetch is not a function".
I updated the configuration to

consumer.run({ autoCommit: true, eachMessage: async ({ topic, partition, message }) => { if (message.headers?.channel) { // manage message } else { // manage message } } });

so that no autocommitThreshold was specified. Same errors "commitOffset is not a function" or "fetch is not a function".

To avoid the issue, the following configuration has been added:

autocommitThreshold: 1

The problem is that the workaround is slowing down the process since the consumer is forced to commit the offset after every message.

Is there any additional configuration to be added to avoid the commit after 1 message and to speed up the process?

Thanks.

@gnpace
Copy link
Author

gnpace commented Nov 5, 2024

Hello,
can anyone help on this?

We’re facing the issue in production environment and we need to fix it as soon as possible.

Thanks for the support.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant