Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

it does not reconnect after failure #48

Open
antst opened this issue Oct 25, 2024 · 2 comments
Open

it does not reconnect after failure #48

antst opened this issue Oct 25, 2024 · 2 comments
Labels

Comments

@antst
Copy link

antst commented Oct 25, 2024

it is easily reproducible.
Just shut down QuestDB for few minutes, and start again. Then QSS component will not attempt reconnects anymore and will not send data, until one restarts home assistant.

Looks like tenacity tries to reconnect for some time and then fails with unhandled exception and then component is dead.
Ideally, I'd assume that it is necessary to handle exception in _retry_data_insertion , and then either skip this attempt to inject data, or keep accumulating data for later send.

Oct 25 16:10:25 hass hass[1814227]: 2024-10-25 16:10:25.141 ERROR (QSS) [root] Uncaught thread exception
Oct 25 16:10:25 hass hass[1814227]: Traceback (most recent call last):
Oct 25 16:10:25 hass hass[1814227]:   File "/opt/homeassistant/current/lib/python3.12/site-packages/tenacity/__init__.py", line 478, in __call__
Oct 25 16:10:25 hass hass[1814227]:     result = fn(*args, **kwargs)
Oct 25 16:10:25 hass hass[1814227]:              ^^^^^^^^^^^^^^^^^^^
Oct 25 16:10:25 hass hass[1814227]:   File "/home/homeassistant/.homeassistant/custom_components/qss/io.py", line 65, in _retry_data_insertion
Oct 25 16:10:25 hass hass[1814227]:     _insert_row_without_auth(host, port, event)
Oct 25 16:10:25 hass hass[1814227]:   File "/home/homeassistant/.homeassistant/custom_components/qss/io.py", line 36, in _insert_row_without_auth
Oct 25 16:10:25 hass hass[1814227]:     with Sender(host, port) as sender:
Oct 25 16:10:25 hass hass[1814227]:   File "src/questdb/ingress.pyx", line 1530, in questdb.ingress.Sender.__enter__
Oct 25 16:10:25 hass hass[1814227]:   File "src/questdb/ingress.pyx", line 1520, in questdb.ingress.Sender.connect
Oct 25 16:10:25 hass hass[1814227]: questdb.ingress.IngressError: Could not connect to "192.168.2.20:9009": No route to host (os error 113)
Oct 25 16:10:25 hass hass[1814227]: The above exception was the direct cause of the following exception:
Oct 25 16:10:25 hass hass[1814227]: Traceback (most recent call last):
Oct 25 16:10:25 hass hass[1814227]:   File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
Oct 25 16:10:25 hass hass[1814227]:     self.run()
Oct 25 16:10:25 hass hass[1814227]:   File "/home/homeassistant/.homeassistant/custom_components/qss/__init__.py", line 156, in run
Oct 25 16:10:25 hass hass[1814227]:     insert_event_data_into_questdb(self.host, self.port, self.auth, event, self.queue)
Oct 25 16:10:25 hass hass[1814227]:   File "/home/homeassistant/.homeassistant/custom_components/qss/io.py", line 70, in insert_event_data_into_questdb
Oct 25 16:10:25 hass hass[1814227]:     _retry_data_insertion(host, port, auth, event)
Oct 25 16:10:25 hass hass[1814227]:   File "/opt/homeassistant/current/lib/python3.12/site-packages/tenacity/__init__.py", line 336, in wrapped_f
Oct 25 16:10:25 hass hass[1814227]:     return copy(f, *args, **kw)
Oct 25 16:10:25 hass hass[1814227]:            ^^^^^^^^^^^^^^^^^^^^
Oct 25 16:10:25 hass hass[1814227]:   File "/opt/homeassistant/current/lib/python3.12/site-packages/tenacity/__init__.py", line 475, in __call__
Oct 25 16:10:25 hass hass[1814227]:     do = self.iter(retry_state=retry_state)
Oct 25 16:10:25 hass hass[1814227]:          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 25 16:10:25 hass hass[1814227]:   File "/opt/homeassistant/current/lib/python3.12/site-packages/tenacity/__init__.py", line 376, in iter
Oct 25 16:10:25 hass hass[1814227]:     result = action(retry_state)
Oct 25 16:10:25 hass hass[1814227]:              ^^^^^^^^^^^^^^^^^^^
Oct 25 16:10:25 hass hass[1814227]:   File "/opt/homeassistant/current/lib/python3.12/site-packages/tenacity/__init__.py", line 419, in exc_check
Oct 25 16:10:25 hass hass[1814227]:     raise retry_exc from fut.exception()
Oct 25 16:10:25 hass hass[1814227]: tenacity.RetryError: RetryError[<Future at 0x7ab2dca52840 state=finished raised IngressError>]
@antst antst added the 🐛 bug label Oct 25, 2024
@CM000n
Copy link
Owner

CM000n commented Nov 18, 2024

Thank you very much @antst. I will have a look at it when I get the chance.
However, I don't have much time at the moment. So any support is very welcome. Feel free to create a PR if you have a direct implementation idea.

@antst
Copy link
Author

antst commented Nov 18, 2024

Actually, I checked, you have already variable RETRY_ATTEMPTS in const.py , which cotrols number of retries. So, for quick fix for me I just set it to very large number ) But, ideally, this needs to be part of config. I will make patch, eventually.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants