MQTT constantly disconnecting and reconnecting. MQTT logs show timeout being exceeded

[masdgaf]

I recently started work on migrating my Home Assistant instance from a HASS OS VM on my homelab VM host to docker containers on a dedicated server. As part of this, I'm moving from the Mosquitto broker add-on to a new Mosquitto docker instance. I have 2 Dreame Z10 Pro vacuums that have been working flawlessly for over a year, but when I enter the info for the new broker, MQTT is dropping and reconnecting constantly. I left both vacuums conencted to the new broker for a few days to see if I could detect a pattern, and it looks like they'll stay connected for 90 seconds, then disconnect and stay that way for about 2 minutes and then reconnect. I've set the broker to allow anonymous connections while troubleshooting to see if that was the issue, but no change. In the Mosquitto logs, I see the following lines:

2022-12-02T18:42:04: Client valetudo_MemorableChiefElephant has exceeded timeout, disconnecting.
...
2022-12-02T18:42:33: New connection from <vacuum IP>:39970 on port 1883.
...
2022-12-02T18:42:33: New client connected from <vacuum IP>:39970 as valetudo_MemorableChiefElephant (p2, c0, k60).

So the keep-alive is set, but I never see PINGREQ/PINGRESP from the vacuums like I am from other devices that are on the broker, and this tracks with the 90 second online time.

The logs from the vacuum are as follows:

2022-12-02T18:51:09.654Z  Info  Attempting to reconnect to MQTT broker
2022-12-02T18:52:45.614Z  Info  Attempting to reconnect to MQTT broker
2022-12-02T18:52:45.750Z  Info  Connected successfully to MQTT broker
2022-12-02T18:52:47.834Z  Info  MQTT configured
2022-12-02T18:54:47.929Z  Info  Connection to MQTT broker closed
2022-12-02T18:54:48.930Z  Info  Attempting to reconnect to MQTT broker
2022-12-02T18:56:21.698Z  Info  Attempting to reconnect to MQTT broker
2022-12-02T18:56:21.798Z  Info  Connected successfully to MQTT broker
2022-12-02T18:56:24.193Z  Info  MQTT configured
2022-12-02T18:57:58.515Z  Info  Connection to MQTT broker closed
2022-12-02T18:57:59.515Z  Info  Attempting to reconnect to MQTT broker
2022-12-02T18:57:59.871Z  Info  Connected successfully to MQTT broker
2022-12-02T18:59:33.624Z  Info  Attempting to reconnect to MQTT broker
2022-12-02T18:59:33.638Z  Info  Connected successfully to MQTT broker
2022-12-02T18:59:35.705Z  Info  MQTT configured
2022-12-02T18:59:36.009Z  Info  MQTT configured
2022-12-02T19:01:35.805Z  Info  Connection to MQTT broker closed
2022-12-02T19:01:36.806Z  Info  Attempting to reconnect to MQTT broker
2022-12-02T19:03:08.643Z  Error  MQTT publication error: Error: Connection closed
    at MqttClient._flushStoreProcessingQueue (/snapshot/Valetudo/node_modules/mqtt/lib/client.js:1892:32)
    at MqttClient.remove (/snapshot/Valetudo/node_modules/mqtt/lib/client.js:1799:12)
    at Object.onceWrapper (node:events:651:28)
    at MqttClient.emit (node:events:549:35)
    at Socket.<anonymous> (/snapshot/Valetudo/node_modules/mqtt/lib/client.js:477:10)
    at Socket.emit (node:events:537:28)
    at TCP.<anonymous> (node:net:747:14)
2022-12-02T19:03:08.645Z  Error  MQTT publication error: Error: Connection closed
    at MqttClient._flushStoreProcessingQueue (/snapshot/Valetudo/node_modules/mqtt/lib/client.js:1892:32)
    at MqttClient.remove (/snapshot/Valetudo/node_modules/mqtt/lib/client.js:1799:12)
    at Object.onceWrapper (node:events:651:28)
    at MqttClient.emit (node:events:549:35)
    at Socket.<anonymous> (/snapshot/Valetudo/node_modules/mqtt/lib/client.js:477:10)
    at Socket.emit (node:events:537:28)
    at TCP.<anonymous> (node:net:747:14)
2022-12-02T19:03:08.647Z  Error  MQTT publication error: Error: Connection closed
    at MqttClient._flushStoreProcessingQueue (/snapshot/Valetudo/node_modules/mqtt/lib/client.js:1892:32)
    at MqttClient.remove (/snapshot/Valetudo/node_modules/mqtt/lib/client.js:1799:12)
    at Object.onceWrapper (node:events:651:28)
    at MqttClient.emit (node:events:549:35)
    at Socket.<anonymous> (/snapshot/Valetudo/node_modules/mqtt/lib/client.js:477:10)
    at Socket.emit (node:events:537:28)
    at TCP.<anonymous> (node:net:747:14)
2022-12-02T19:03:09.642Z  Info  Attempting to reconnect to MQTT broker

It doesn't seem to be the connection issues that other users have seen relating to network problems, and I have multiple other devices that are using this broker with no issues, so it doesn't seem to be a misconfiguration there. I have tried putting the vacuums back on the old broker and they stay connected there no problem. Any help with this would be greatly appreciated.

[masdgaf]

I'm still trying to figure out what the issue is on this. I tried running Mosquitto directly on the Debian host thinking that would remove any potential issues introduced by using Docker, but am still seeing the exact same behavior with these lines appearing repeatedly in the Mosquitto log file:

2022-12-12T16:39:19: New connection from <vacuum IP>:40972 on port 1883.
2022-12-12T16:39:19: New client connected from <vacuum IP>:40972 as valetudo_MemorableChiefElephant (p2, c0, k60, u'mqttuser').
2022-12-12T16:40:54: Client valetudo_MemorableChiefElephant has exceeded timeout, disconnecting.

Again, this behavior is happening with 2 vacuums.

[masdgaf]

I got around this issue by spinning up a new Mosquitto instance on another Docker host with the exact same config, moving the vacuums to that broker for testing, and then moving them back to the broker referenced here again. No idea why that would fix the issue, but there you go.