Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide documentation how to handle producer backpressure in Pulsar client applications #40

Open
lhotari opened this issue May 23, 2023 · 0 comments

Comments

@lhotari
Copy link

lhotari commented May 23, 2023

When the client application produces messages faster than the Pulsar broker (Astra Streaming service) can accept, the broker will create back pressure on message sending. It is the responsibility of the client application to handle this.

Astra Streaming will throttle by default:
https://docs.datastax.com/en/streaming/astra-streaming/operations/astream-limits.html

The client application cannot assume that the broker has accepted the message for delivery until the send operation completes successfully. It isn't a data loss caused by our service when the broker throttles message sending. The client application must handle this type of scenario.

Based on our experience, one possible way to handle back pressure for a Pulsar producer is to set the Pulsar client's producer send timeout setting to infinite or a high value, and to enable the blockIfQueueFull setting.

When looking into this, I noticed that in the Apache Pulsar community, this type of knowledge hasn't been well-documented or shared.

How are our customers supposed to know that client applications are responsible for handling back pressure?
How could they understand what actions are necessary to manage back pressure properly?

The expected outcome of addressing this issue is to have answers to the above questions.
This documentation should be also contributed to Apache Pulsar upstream documentation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant