How and where best to handle errors related to connection loss to Zino server

[podliashanyk]

Per bug described in #94: Uninett/zinolib#59 makes error message more user friendly yet the problem itself is not yet fixed.

Questions:

• Where does the fix belong? zinolib, Howitz, deployment config, a combination?
• How should we approach fixing the problem?

Briefly on current authentication in Howitz:

• using flask-login
• both Flask Session and Zino server authentication must be OK for user to be considered authenticated. If at least 1 of them in not OK the current app behaviour is to redirect to /login

Related issues:

• When connection to Zino server is temporarily lost, Howitz stops working until restarted #94
• Raise better exception if losing connection zinolib#59
• Reconnect does not work after lost connection zinolib#60

[podliashanyk]

The ideal solution would require minimum action from the user. So fixing the issue without redirect to login is a basis I think.

[podliashanyk]

And actually an occasional short-time disconnect should ideally require no action from a user whatsoever

[hmpf]

The username should already be in the session so we could just look up the token directly and relogin behind the scenes.

[stveit]

If the disconnect can be "felt" and depending on how long the relogging takes, some sort of "reconnecting" animation/message could be useful

[podliashanyk]

BrokenPipe error still needs fixing without Flask app restart.
If I understand correctly, it is zinolib losing connection to Zino server that causes the problem. BrokenPipe error is caught in zinolib. Should zinolib handle it (reconnect and fixing connection data back to normal)? Then Howitz probably wouldn't need "relogin behind the scenes" at all.

[podliashanyk]

If the disconnect can be "felt" and depending on how long the relogging takes, some sort of "reconnecting" animation/message could be useful

Probably a special kind of error message, like a red banner below the navbar. Argus-style :)

[hmpf]

I have a branch in zinolib that catches BrokenPipe in one place we know it can happen, and translates it to something easier to understand for a human (LostConnectionError).

If there is a global exception handler, if it can catch and treat LostConnectionError and BrokenPipe as the same thing and relogin the user behind the scenes, that should be suffcient I think.

[hmpf]

The traceback, if it is BrokenPipe, still needs to be logged, as we want to translate from BrokenPipe to LostConnectionError everywhere it is provably triggered. Might have to catch it in zinolib.ritz, even.

If it is LostConnection we can just log it without traceback so we can see how common it is to lose the connection.

[hmpf]

If the relogin takes a while we could show the user "There is a glitch in the matrix, please hold" or something as a fun error message.

[podliashanyk]

If there is a global exception handler, if it can catch and treat LostConnectionError and BrokenPipe as the same thing and relogin the user behind the scenes, that should be suffcient I think.

Have attempted it but it is not enough.
I have reconnected to Zino server, and was afterwards getting True for both is_authenticated and is_connected from the EventManager. Yet every request after the reconnect was still failing with either NotConnectedError, LostConnectionError or BrokenPipeError

[podliashanyk]

Bug report in zinolib Uninett/zinolib#60

[podliashanyk]

PR with fix for the problem with having to restart the app completely when losing connection to Zino server: #110

[podliashanyk]

Created #115

[podliashanyk]

Created #116

[hmpf]

Not so much a bug as a feature that howitz has not copied from curitz.

[hmpf]

With zinolib 1.2.0, on running manager.test_connection() we will now get another source of TimeoutError.

I tried running the test just before the for-loop in howitz.endpoints.refresh_current_events but that runs the test every refresh_interval seconds which currently is a bit often. The net can come back even after more than ten seconds have passed =)

[hmpf]

We should find a way to run test_connection that is independent of refresh_interval.

Then there's the matter of what to do on a timeout. Show a fail-whale? Retry after 5, 10 minutes or on manual reload of page?

[podliashanyk]

Like the idea to run an independent test_connection!

What about something like this: on timeout --> retry max_retry times every timeout s (meanwhile show red bar below navbar with err msg a-la "Lost connection to server. Attempting reconnect" --> show a fail-whale if all retries fail

[hmpf]

Okay, I just heard from @runborg that the zino 1 server has a problem with it's ntie sockets. There's no keep-alive, which means that with for instance the right/wrong firewall between the server and the clients, the ntie socket will quietly die regardless.

We've quickly sketched out several mitigations:

• the zino 1 server regularly sending unhandled garbage updates
• making a keep-alive update type
• the clients sending something on the ntie socket, but this might choke the server eventually

Regardless, something will need to change in the actual server.

[podliashanyk]

Making a keep-alive update type is best IMO

[podliashanyk]

Related issue in zinolib Uninett/zinolib#62

[hmpf]

See Uninett/zinolib#63