feat: Secure document encryption key exchange

[islamaliev]

Relevant issue(s)

Resolves #2856 #2909

Description

This PR introduces a secure mechanism for exchanging document encryption keys in our decentralized network. It supports both whole-document and field-level encryption, enhancing data privacy without compromising the system's distributed nature.

Data Flow and Key Exchange Process

1. Document Creation:

   • When a user creates a document with encryption enabled (whole document or specific fields), the system generates a new DAG block containing the encrypted delta.
   • A new Encryption IPLD block is created, containing the docID, optional fieldName (for field-level encryption), and the encryption key itself.
   • The DAG block references the Encryption block by storing its CID.

2. Encryption Detection:

   • Upon receiving an update event for a document, a node checks if the block is encrypted by examining the reference to the Encryption block.
   • If encrypted and the node lacks the corresponding key, it initiates a key fetch request.

3. Key Request and Retrieval:

   • The node sends an "enc-keys-request" event on the event bus, which is picked up by the Key Management Service (KMS).
   • The KMS handles the key exchange process over the "encryption" pubsub channel.

4. Key Reception and Storage:

   • Upon receiving the encryption keys, the KMS stores them in a dedicated IPLD storage.
   • It then notifies the merging process that is waiting for the results.

5. Decryption and Merging:

   • During the executeMerge process, the node collects CIDs of all Encryption blocks that are not available locally.
   • After the initial merge attempt, if any encryption keys are missing, the node requests these keys via the KMS.
   • Once the keys are received, the merge process is restarted to decrypt and apply the previously encrypted deltas.

Technical Details

• Encryption: ECIES with X25519, HKDF (SHA-256), and AES-256-GCM
• Key Storage: Encryption keys are stored in separate IPLD blocks, referenced by the main DAG blocks
• Key Management: Introduced a new KMS abstraction for handling key exchange and storage
• Pubsub: All peers listen on the "encryption" topic for key exchange messages
• Event Bus: New "enc-keys-request" event for initiating key requests

Current Limitations and Future Work

• This version assumes all documents are public. Future iterations will integrate with access control mechanisms.
• The system currently trusts the first received response. Future enhancements will implement more robust validation.
• Further work is needed to optimize the key exchange process for large-scale deployments and to handle network partitions or failed exchanges gracefully.
• The current KMS implementation can be replaced in the future, e.g., with an Orbis KMS, for more advanced key management features.

Testing Improvements

Introduced an "assert stack" for integration tests, providing detailed failure contexts (e.g., 'path: commits[2].links[1].cid' instead of 'doc: 1').

These changes significantly enhance our system's security, enabling confidential data storage and transmission in a decentralized environment, while laying the groundwork for more advanced encryption and access control features. The introduction of the KMS abstraction provides flexibility for future improvements in key management.

Tasks

• I made sure the code is well commented, particularly hard-to-understand areas.
• I made sure the repository-held documentation is changed accordingly.
• I made sure the pull request title adheres to the conventional commit style (the subset used in the project can be found in tools/configs/chglog/config.yml).
• I made sure to discuss its limitations such as threats to validity, vulnerability to mistake and misuse, robustness to invalidation of assumptions, resource requirements, ...

How has this been tested?

With integration tests.

Specify the platform(s) on which this was tested:

• MacOS

[fredcarle]

Initial batch of comment bellow. There are a few parts I want to go over again but overall it looks really nice.

[fredcarle]

LGTM. Just a couple todos to resolve before merge. Thanks Islam!

[fredcarle]

todo: It doesn't make sense to create a dependency to the net package only for this. You can even create the grpcpeer.Peer struct inline if you want.

[islamaliev]

done

[fredcarle]

todo: addPubSubTopic already exists. You can simply call it from this function if you need it to be public.

Took into consideration all comments and got approval from Fred. Any further improvement can be handled later.