Production : Solace PubSub+ connector version 1.3.0 Connector Issue : Connection breaks with broker in subscribing API

[teredpus]

We noticed that with latest versions of solace pubsub+ connector 1.3.0 , the consuming API connection gets interrupted with solace queue and messages gets start piling up in solace queues.

Once the API restarts the connector reconnects and starts consuming the messages.
Connector configurations have reconnection strategy set to forever.

In old version, this issue wasn't observed. We have observed this behavior with new version.

We have proper reconnection strategy configured and have error handling in place.
This behavior is observed for multiple APIs.

[sunil-solace]

@teredpus the default reconnection retries by JCSMP client is now set to 0 since v1.2.0 https://github.com/SolaceProducts/pubsubplus-connector-mule-docs/blob/main/doc/release-notes.md#v120. Hence as you said you're configuring the Mule Reconnection Strategy forever, can you share the logs of the issue.
Also can you elaborate what do you mean by multiple APIs.

[teredpus]

Here the MuleSoft logs does not have any traces of issue but suddenly the api consumer disappears from solace queue and then messages starts piling up. Only after MuleSoft api restarts, it reconnects and consumes all messages.

[sunil-solace]

@teredpus as I mentioned before that the default reconnection strategy is at the connector client level is disabled, the connector will no more try to reconnect unless you've configured it to. You can read more about it in this doc https://github.com/SolaceProducts/pubsubplus-connector-mule-docs/blob/main/doc/technical-reference.md#reconnection .

I hope you have implemented either of the mentioned Reconnection strategy here which will log if it is really trying to reconnect. Hence, the logs are crucial here. Moreover you can enable root level trace logging and share with us, which can help to root cause the problem.

Because we're not seeing any such issue, you can also share a sample Mulesoft Flow and steps to reproduce this scenario that you're talking about, which we can then try to reproduce on our end.

[teredpus]

We have reconnection strategy set to forever with frequency.

Its our observation that after seeing below error , the connection breaks and solace queues starts piling up.

You can have reference to petcare-onetrust-sys-pr API, this API is using 1.3.1 version, have attached stack trace below :

03:50:31.244 07/28/2023 Worker-0 Context_3_ConsumerDispatcher ERROR

---

Message : (JCSMPTransportException) Error receiving data from underlying connection.
Element : (None)
Element DSL : (None)
Error type : (None)
FlowStack : (None)

Root Exception stack trace:
org.mule.runtime.api.connection.ConnectionException: (JCSMPTransportException) Error receiving data from underlying connection.
at com.solace.connector.mulesoft.internal.jcsmp.SolaceMessageListener.onException(SolaceMessageListener.java:125)
at com.solacesystems.jcsmp.protocol.nio.impl.ConsumerErrorNotification.handleNotification(ConsumerErrorNotification.java:40)
at com.solacesystems.jcsmp.protocol.nio.impl.ConsumerNotificationDispatcher.eventLoop(ConsumerNotificationDispatcher.java:112)
at com.solacesystems.jcsmp.protocol.nio.impl.ConsumerNotificationDispatcher.run(ConsumerNotificationDispatcher.java:130)
at java.lang.Thread.run(Thread.java:750)

---

[sunil-solace]

For now can you disable the Mulesoft's reconnection strategy and enable the JCSMP reconnection config. ie. CLIENT_CHANNEL_PROPERTIES.ReconnectRetries and see if this solves your problem for now. We will further look on our end if there's any corner case to Mule reconnection strategy is not triggered in some scenario.