Client stops trying to connect if disconnected_cb raises exception

[mprimi]

If the callback passed to disconnected_cb raises an exception, then the NATS client will never reconnect to the cluster.

While this behavior could be classified as "user error" (that callback should not throw!), it would be nice to handle this gracefully:

• Print an error message so the user knows the callback has raised an exception (currently no message)
• Keep trying to reconnect (currently, no attempt to reconnect)

Attaching a simple script that reproduces the behavior: callback_exception.py.txt

See instructions at the top of the file.

Expected behavior:

📤 Message published
📥 Message received
📤 Message published
📥 Message received
📤 Message published
📥 Message received
>>>> Server crash-restart
nats: encountered error
NoneType: None
🔴 Disconnected!
🟢 Reconnected!
📤 Message published
📥 Message received
📤 Message published
📥 Message received

Actual behavior (with DISCONNECT_CALLBACK_EXCEPTION=True):

📤 Message published
📥 Message received
📤 Message published
📥 Message received
📤 Message published
📥 Message received
>>>> Server crash-restart
nats: encountered error
NoneType: None
🔴 Disconnected!
📤 Message published
📤 Message published
📤 Message published
📤 Message published

Notice no errors, no attempts to reconnect, and no message delivered forever after

[mprimi]

Possible solution discussed in Slack: wrap all callbacks so they can't do any damage if they raise exceptions

[theobouwman]

Would this potentially solve airtai/faststream#1581 (comment) ?