Data corruption protection

[a3kov]

Why

I think this is a fantastic library and a game changer for the BEAM world!
In order to get more trust in mission-critical environments I think it would be cool for it to have mechanism to protect itself from silent data corruptions and ability to self-heal automatically.
I understand Raft provides some degree of protection, but does it prevent it completely ? I don't think so

How

Perhaps, it should use check sums ? Periodic checks ? Verify check sum on every read ?
Obviously such feature will have performance penalty and should be optional.

[michaelklishin]

Raft does not include checksumming. A way to store checksums with keys and values in Khepri would be nice but this needs to be investigated first => moving this to a discussion.

[dumbbell]

For the record, Ra supports checksums in its write-ahead log (optional, disabled by default), in segment files (optional, enabled by default) and in snapshots (always enabled).

[a3kov]

Yeah, I meant there's a degree of protection initially thanks to replication.

[dumbbell]

What kind of corruptions would you like Khepri to detect/handle? Like, problems with disk storage, problems with memory hardware, bugs in Erlang/Raft/Khepri, etc.

[a3kov]

Ideally, all kinds :)
Most obvious probably would be storage-related, as data could become corrupted on replicas, and reads are local.

[a3kov]

There's also question of scope - I understand you built it to store (small) metadata in Rabbitmq, but if it could store data reliably, it could become the recommended general purpose db in the BEAM ecosystem. The more data you store, the higher is the standard.
So if this feels like out of scope for this library - just ignore me

[dumbbell]

Ra already uses checksums for its segment files and snapshots (enabled by default). It can also use checksums for its write-ahead log (disabled by default). It covers error detection for on-disk corruption. I don't remember if a Ra server can re-fetch data from other cluster members in case of a bad checksum. If it doesn't do that already, it would be a nice feature! However, for a single node, it doesn't handle healing for sure, but I feel it's the responsibility of the underlying filesystem/volume and/or the backup strategy.

For in-memory corruptions, I also feel it would be too much for Khepri. It would probably be more appropriate to handle that at a lower level (Erlang, OS, ECC memory).

My general feeling is that this corruption protection seems out-of-scope for Khepri. Some bits are handled by Ra. That plus backups seem to cover a lot of ground already. That said, I still want to hear more from you, especially if you had an example of a situation where you missed such a protection/healing ability. This would help me think this through a lot more.

[a3kov]

I personally haven't encountered it. I just thought that it would be a cool feature. But if Ra has it already - that's great!
I think with ECC RAM it should be pretty rare, and disks usually have some sort of https://en.wikipedia.org/wiki/Error_correction_code implemented in hardware/firmware.
A feature like this could help gain more adoption.

[michaelklishin]

Yeah, I meant Raft the protocol/paper, Ra does offer some checksum features and we have seen their effects in practice with RabbitMQ.

That should be enough for more than 90%. For others, it may be a good idea to checksum and verify the values stored in Khepri. What is not obvious to me right now is what can be done (besides relying on Ra, that is) to check sum the keys as well. Note that fetching a single value by key does not need this but when you get a list of keys back, you cannot be certain that exactly the requested key was loaded.

Corruptions of keys specifically is a very rare scenario but some storage systems, IIRC Cassandra, care about key integrity as much as about that of values.