Error in walker

[hgn]

It seems that it is an error in the walker. After adding tests with multiple categories (and at least one crash before caused by invalid data) the following error log shows up and hippod crashes after each reboot (which should never ever happens). Please follow the error path and implement a proper error handling and catches errors like "key is not available", array size is 0 where you "normally" expected != 0. Here is the error log:

starting remove old and unnecessary files
    Excuted in 6.57 seconds
  Exectute Garbage Collector for MimeDB
    Excuted in 0.13 seconds
Traceback (most recent call last):
  File "./run.py", line 352, in <module>
    main(conf)
  File "./run.py", line 276, in main
    register_timeout_handler(app)
  File "./run.py", line 269, in register_timeout_handler
    register_timeout_handler_daily(app)
  File "./run.py", line 263, in register_timeout_handler_daily
    timeout_daily(app)
  File "./run.py", line 241, in timeout_daily
    cache_update_daily(app)
  File "./run.py", line 212, in cache_update_daily
    cache_achievement = cache_achievements.Cache(app, frequency)
  File "/home/pfeifer/src/code/misc/hippod/hippod/cache_achievements.py", line 14, in __init__
    self.stored_data = self.read_old_cache()
  File "/home/pfeifer/src/code/misc/hippod/hippod/cache_achievements.py", line 48, in read_old_cache
    return self.data['achievements-by-time']
KeyError: 'achievements-by-time'

Before I added data to dataset which where probably corrupt, but this should be catched by the object post routine: corrupt data should never ever go into the data, here is the trace:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/aiohttp/server.py", line 265, in start
    yield from self.handle_request(message, payload)
  File "/usr/local/lib/python3.5/dist-packages/aiohttp/web.py", line 96, in handle_request
    resp = yield from handler(request)
  File "/home/pfeifer/src/code/misc/hippod/hippod/aiohttp_index.py", line 47, in index_handler
    return await handler(request)
  File "/home/pfeifer/src/code/misc/hippod/hippod/api_object_post.py", line 407, in handle
    data = try_adding_xobject(app, xobj)
  File "/home/pfeifer/src/code/misc/hippod/hippod/api_object_post.py", line 355, in try_adding_xobject
    sha_major, sha_minor = hippod.hasher.check_sum_object_issue(xobj['object-item'])
  File "/home/pfeifer/src/code/misc/hippod/hippod/hasher.py", line 60, in check_sum_object_issue
    majorbuf = ''.join(o["title"]) + ''.join(o["categories"])
TypeError: sequence item 0: expected str instance, list found

So there are two problems that needs to be fixed.

[hgn]

You can trigger the problem when you run tests and interrupt the daemon (CTRL-C) after some time. Then restart the server via sudo ./run.py -f /etc/hippod/hippod-configuration.json and the error should pop off.

[hgn]

This also happens when I interrupt the process during startup (garbage collection).

[hgn]

New info: it seems that the error is triggered more often then expected. The stack trace looks like:

Traceback (most recent call last):
  File "./run.py", line 352, in <module>
    main(conf)
  File "./run.py", line 276, in main
    register_timeout_handler(app)
  File "./run.py", line 269, in register_timeout_handler
    register_timeout_handler_daily(app)
  File "./run.py", line 263, in register_timeout_handler_daily
    timeout_daily(app)
  File "./run.py", line 241, in timeout_daily
    cache_update_daily(app)
  File "./run.py", line 212, in cache_update_daily
    cache_achievement = cache_achievements.Cache(app, frequency)
  File "/home/pfeifer/src/code/misc/hippod/hippod/cache_achievements.py", line 14, in __init__
    self.stored_data = self.read_old_cache()
  File "/home/pfeifer/src/code/misc/hippod/hippod/cache_achievements.py", line 48, in read_old_cache
    return self.data['achievements-by-time']
KeyError: 'achievements-by-time'

and I think the walker runs more often as expected. It should run daily, but it seems it runs more often. Interrupting it will lead to corrupted database. As I said before:

1. the database should never exhibit incorrect/corrupt data. Why is the data corrupt?
2. the daily runner should excecute daily and at program start, not more often.
3. The keyerror is a consequence of problem 1. anyway, this code should also be aware that the database is somehow corrupt (print error message like database corrupt) AND additionally catch the error in some way.

[srmnw]

The first problem is not really one. I've just accidentally returned a wrong variable.
The second error occurs very probably when there is no string in title/category but a list. I will fix that.