improve monitoring of sync service

[jas4711]

It would be nice to monitor how syncing works -- whether two servers are fully in sync or not.

[dainnilsson]

We currently have two utilities for checking sync: ykval-checksum-clients and ykval-checksum-deactivated.
These generate a checksum over different parts of the database: the clients, and the yubikeys (where active = false) tables.

We could add a ykval-checksum-activated which does the same but for active keys. This command could be used to compare the state of different servers. The problem being that the state can change rather quickly, and synchronization between servers is usually delayed by several seconds, so getting something meaningful out of this might be difficult as I can image under large scale use two servers are hardly ever fully in sync.

[klali]

I don't think we want to reuse the same system of checksumming for monitoring sync, it's to common that it isn't exactly in sync. The simplest approach is probably to just monitor queue size, though some errors can slip past that, so we might want to think up something more clever..

[jas4711]

We thought about this problem and believe that we should have some software running on each api server to monitor this in real-time. Not working on this now...