Bubuku dies on any exception and terminates kafka instance

[antban]

The problem is that if bubuku is in docker, than kafka instance will be terminated immediately (without writing data).

In our case it was like this:

Aug 19 10:58:07 ip-172-31-139-127 docker/26f64e7b1654[888]: WARNING:kazoo.client:Connection dropped: socket connection error: None
Aug 19 10:58:07 ip-172-31-139-127 docker/26f64e7b1654[888]: INFO:kazoo.client:Connecting to 172.31.172.94:2181
Aug 19 10:58:07 ip-172-31-139-127 docker/26f64e7b1654[888]: WARNING:kazoo.client:Connection dropped: socket connection broken
Aug 19 10:58:08 ip-172-31-139-127 docker/26f64e7b1654[888]: INFO Opening socket connection to server ip-172-31-174-143.eu-west-1.compute.internal/172.31.174.143:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
Aug 19 10:58:08 ip-172-31-139-127 docker/26f64e7b1654[888]: Traceback (most recent call last):
Aug 19 10:58:08 ip-172-31-139-127 docker/26f64e7b1654[888]:   File "/usr/local/lib/python3.5/dist-packages/kazoo/retry.py", line 123, in __call__
Aug 19 10:58:08 ip-172-31-139-127 docker/26f64e7b1654[888]:     return func(*args, **kwargs)
Aug 19 10:58:08 ip-172-31-139-127 docker/26f64e7b1654[888]:   File "/usr/local/lib/python3.5/dist-packages/kazoo/client.py", line 1026, in get
Aug 19 10:58:08 ip-172-31-139-127 docker/26f64e7b1654[888]:     return self.get_async(path, watch).get()
Aug 19 10:58:08 ip-172-31-139-127 docker/26f64e7b1654[888]:   File "/usr/local/lib/python3.5/dist-packages/kazoo/handlers/utils.py", line 72, in get
Aug 19 10:58:08 ip-172-31-139-127 docker/26f64e7b1654[888]:     raise self._exception
Aug 19 10:58:08 ip-172-31-139-127 docker/26f64e7b1654[888]: kazoo.exceptions.SessionExpiredError
Aug 19 10:58:08 ip-172-31-139-127 docker/26f64e7b1654[888]: 
Aug 19 10:58:08 ip-172-31-139-127 docker/26f64e7b1654[888]: During handling of the above exception, another exception occurred:
Aug 19 10:58:08 ip-172-31-139-127 docker/26f64e7b1654[888]: 
Aug 19 10:58:08 ip-172-31-139-127 docker/26f64e7b1654[888]: Traceback (most recent call last):
Aug 19 10:58:08 ip-172-31-139-127 docker/26f64e7b1654[888]:   File "/usr/local/bin/bubuku", line 11, in <module>
Aug 19 10:58:08 ip-172-31-139-127 docker/26f64e7b1654[888]:     sys.exit(main())
Aug 19 10:58:08 ip-172-31-139-127 docker/26f64e7b1654[888]:   File "/usr/local/lib/python3.5/dist-packages/bubuku/daemon.py", line 81, in main
Aug 19 10:58:08 ip-172-31-139-127 docker/26f64e7b1654[888]:     controller.loop()
Aug 19 10:58:08 ip-172-31-139-127 docker/26f64e7b1654[888]:   File "/usr/local/lib/python3.5/dist-packages/bubuku/controller.py", line 125, in loop
Aug 19 10:58:08 ip-172-31-139-127 docker/26f64e7b1654[888]:     self.make_step(ip)
Aug 19 10:58:08 ip-172-31-139-127 docker/26f64e7b1654[888]:   File "/usr/local/lib/python3.5/dist-packages/bubuku/controller.py", line 143, in make_step
Aug 19 10:58:08 ip-172-31-139-127 docker/26f64e7b1654[888]:     self._add_change_to_queue(check.check_if_time())
Aug 19 10:58:08 ip-172-31-139-127 docker/26f64e7b1654[888]:   File "/usr/local/lib/python3.5/dist-packages/bubuku/controller.py", line 37, in check_if_time
Aug 19 10:58:08 ip-172-31-139-127 docker/26f64e7b1654[888]:     return self.check()
Aug 19 10:58:08 ip-172-31-139-127 docker/26f64e7b1654[888]:   File "/usr/local/lib/python3.5/dist-packages/bubuku/features/restart_if_dead.py", line 58, in check
Aug 19 10:58:08 ip-172-31-139-127 docker/26f64e7b1654[888]:     if self.broker.is_running_and_registered():
Aug 19 10:58:08 ip-172-31-139-127 docker/26f64e7b1654[888]:   File "/usr/local/lib/python3.5/dist-packages/bubuku/broker.py", line 29, in is_running_and_registered
Aug 19 10:58:08 ip-172-31-139-127 docker/26f64e7b1654[888]:     return self.id_manager.is_registered()
Aug 19 10:58:08 ip-172-31-139-127 docker/26f64e7b1654[888]:   File "/usr/local/lib/python3.5/dist-packages/bubuku/id_generator.py", line 62, in is_registered
Aug 19 10:58:08 ip-172-31-139-127 docker/26f64e7b1654[888]:     return self.zk.is_broker_registered(self.broker_id)
Aug 19 10:58:08 ip-172-31-139-127 docker/26f64e7b1654[888]:   File "/usr/local/lib/python3.5/dist-packages/bubuku/zookeeper/__init__.py", line 166, in is_broker_registered
Aug 19 10:58:08 ip-172-31-139-127 docker/26f64e7b1654[888]:     _, stat = self.exhibitor.get('/brokers/ids/{}'.format(broker_id))
Aug 19 10:58:08 ip-172-31-139-127 docker/26f64e7b1654[888]:   File "/usr/local/lib/python3.5/dist-packages/bubuku/zookeeper/__init__.py", line 110, in get
Aug 19 10:58:08 ip-172-31-139-127 docker/26f64e7b1654[888]:     return self.client.retry(self.client.get, *params)
Aug 19 10:58:08 ip-172-31-139-127 docker/26f64e7b1654[888]:   File "/usr/local/lib/python3.5/dist-packages/kazoo/client.py", line 273, in _retry
Aug 19 10:58:08 ip-172-31-139-127 docker/26f64e7b1654[888]:     return self._retry.copy()(*args, **kwargs)
Aug 19 10:58:08 ip-172-31-139-127 docker/26f64e7b1654[888]:   File "/usr/local/lib/python3.5/dist-packages/kazoo/retry.py", line 136, in __call__
Aug 19 10:58:08 ip-172-31-139-127 docker/26f64e7b1654[888]:     raise RetryFailedError("Exceeded retry deadline")
Aug 19 10:58:08 ip-172-31-139-127 docker/26f64e7b1654[888]: kazoo.retry.RetryFailedError: Exceeded retry deadline

In case of any unpredictable errors bubuku should reinitialize itself without terminating kafka, and continue to work.