Python/pytest hide exceptions that are thrown while logging

[Trolldemorted]

If we

• invoke pytest with -W error, python will raise exceptions for warnings as it should
• if those warnings occur during logging, we neither get a warning, nor a failing test, because something decides that exceptions during logging just cause "no logs for you", and not a failure

We should find out what exactly is happening, and consider raising an upstream issue.

cc @ldruschk