You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When the client application produces messages faster than the Pulsar broker (Astra Streaming service) can accept, the broker will create back pressure on message sending. It is the responsibility of the client application to handle this.
The client application cannot assume that the broker has accepted the message for delivery until the send operation completes successfully. It isn't a data loss caused by our service when the broker throttles message sending. The client application must handle this type of scenario.
Based on our experience, one possible way to handle back pressure for a Pulsar producer is to set the Pulsar client's producer send timeout setting to infinite or a high value, and to enable the blockIfQueueFull setting.
When looking into this, I noticed that in the Apache Pulsar community, this type of knowledge hasn't been well-documented or shared.
How are our customers supposed to know that client applications are responsible for handling back pressure?
How could they understand what actions are necessary to manage back pressure properly?
The expected outcome of addressing this issue is to have answers to the above questions.
This documentation should be also contributed to Apache Pulsar upstream documentation.
The text was updated successfully, but these errors were encountered:
When the client application produces messages faster than the Pulsar broker (Astra Streaming service) can accept, the broker will create back pressure on message sending. It is the responsibility of the client application to handle this.
Astra Streaming will throttle by default:
https://docs.datastax.com/en/streaming/astra-streaming/operations/astream-limits.html
The client application cannot assume that the broker has accepted the message for delivery until the send operation completes successfully. It isn't a data loss caused by our service when the broker throttles message sending. The client application must handle this type of scenario.
Based on our experience, one possible way to handle back pressure for a Pulsar producer is to set the Pulsar client's producer send timeout setting to infinite or a high value, and to enable the blockIfQueueFull setting.
When looking into this, I noticed that in the Apache Pulsar community, this type of knowledge hasn't been well-documented or shared.
How are our customers supposed to know that client applications are responsible for handling back pressure?
How could they understand what actions are necessary to manage back pressure properly?
The expected outcome of addressing this issue is to have answers to the above questions.
This documentation should be also contributed to Apache Pulsar upstream documentation.
The text was updated successfully, but these errors were encountered: