Sending messages with confirmation to a full queue of limited size can lead to high channel churn

[groundhog2k]

Scenario:
On RabbitMQ side an exchange and a queue was configured. The queue has a binding to the exchange.
The queue has a limited size and an overflow behavior "reject-publish".

A spring based java application client is sending messages to the exchange using the rabbitmq-java-client library and waits for confirmation of every sent message - or until a timeout happens.
When the queue is full the client will receive "nack" messages as exceptions.

The problem is, that under the hood (inside the rabbitmq-java-client implementation), every "Queue full" response from RabbitMQ will lead to a closed channel.
This can be seen in the code at: https://github.com/rabbitmq/rabbitmq-java-client/blob/main/src/main/java/com/rabbitmq/client/impl/ChannelN.java#L241
It happens exactly in Line 248.

When the message rate of published messages is very high, it causes the effect that a lot of new channels will be opened and closed - and Pooled channels can not be reused anymore, because they were already closed by mentioned line of code.
That all ends in high channel churn which is described in https://www.rabbitmq.com/channels.html#high-channel-churn and also pooling of channels will not work anymore.
High channel churn can then lead to more resource usage (memory, cpu) on RabbitMQ side and so on...

Is there a better way to differentiante between the "queue full" case and other potential errors and handle the different cases better inside https://github.com/rabbitmq/rabbitmq-java-client/blob/main/src/main/java/com/rabbitmq/client/impl/ChannelN.java#L241.

In my opinion it is not necessary to close a channel in case of "queue full" - it can still be reused.

This affects all versions of the rabbitmq-java-client.

[michaelklishin]

The method in question is waitForConfirmsOrDie, we highly do not recommend that option, and in particular in environments where nacks are likely.

Use Streaming confirms, they won't result in channel closures.

[acogoluegnes]

There's nothing much we can do as long as you use the Channel#waitForConfirmsOrDie(long) method. Its name and contract are clear. See the publisher confirms overview that Michael pointed out and the publisher confirms tutorial for more details.

[groundhog2k]

When this method (Channel#waitForConfirmsOrDie(long)) is highly not recommended for usage then I would suggest to remove it in a later version and let the consumer of the API decide whether it is useful to close a channel or not.

A bit more background:
At the moment our project is using apache-camel which has a component called camel-rabbitmq.
This thing is calling waitForConfirmsOrDie() in https://github.com/apache/camel/blob/camel-3.20.x/components/camel-rabbitmq/src/main/java/org/apache/camel/component/rabbitmq/RabbitMQMessagePublisher.java#L162
So in a high load scenario with a full queue it will receive "nacks" and starts closing channels. So all channels in a channel pool get unusable and it starts opening and closing new channels rapidly. (=> high channel churn)
I think beside my suggestion here, I will also open a bug for the camel-rabbitmq component. Because this is the root cause of the problem. Removing the method here in the rabbitmq-java-client could prevent wrong usage in future implementations.

Btw.: I looked at newer versions of spring (spring-amqp) and they register a ConfirmListener and catch the ack/nack confirmations but do not touch the channel afterwards. (Example: https://github.com/spring-projects/spring-amqp/blob/main/spring-rabbit/src/main/java/org/springframework/amqp/rabbit/connection/PublisherCallbackChannelImpl.java)

[michaelklishin]

The "OrDie" part of the method tells that if there is a timeout or nack, the channel will be closed.
What you are looking at is by design.

[michaelklishin]

What Spring AMQP does is their decision. This method has been around for more than a decade. There is an alternative way of working with confirms, including nacks, and has been for many years.

We will not be changing this method in the Java client. Use streaming confirms.