Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Secure document encryption key exchange #2891

Merged
Merged
Changes from 1 commit
Commits
Show all changes
101 commits
Select commit Hold shift + click to select a range
8a4b322
Update protobuf
islamaliev Jul 13, 2024
8c81b46
Fix after rebase
islamaliev Jul 17, 2024
b6ef757
Enable very naive key exchange
islamaliev Jul 24, 2024
5d18970
Add protobuf data for key request/response
islamaliev Jul 27, 2024
83cb8da
Encrypt peer-to-peer data exchange
islamaliev Jul 27, 2024
3153090
Merge remote-tracking branch 'upstream/develop' into feat/encryption-…
islamaliev Jul 27, 2024
c2ff5a3
Doc field key exchange
islamaliev Jul 30, 2024
1505ace
Decrypt doc-level and field-level block simultaneously
islamaliev Aug 1, 2024
4782ea5
Add encryption with ECDH
islamaliev Aug 3, 2024
5d83085
Remove unnecessary pub key transit
islamaliev Aug 3, 2024
cf5c559
Implement ECIES
islamaliev Aug 3, 2024
d46b881
Adjustments
islamaliev Aug 3, 2024
4eafe57
Polish
islamaliev Aug 4, 2024
abb3f99
Remove unnecessary peerInfo transit
islamaliev Aug 4, 2024
58bf7ce
Polish
islamaliev Aug 4, 2024
9d89392
Merge remote-tracking branch 'upstream/develop' into feat/encryption-…
islamaliev Aug 4, 2024
a24c9b0
Make all mission enc keys be batched
islamaliev Aug 5, 2024
31e0c37
Load latest available encrypted block from Blockstore instead of fetc…
islamaliev Aug 5, 2024
809e181
Make block store height of where encryption started
islamaliev Aug 7, 2024
a127298
Make failed tests show this 'path: commits[2].links[1].cid' instead o…
islamaliev Aug 7, 2024
f62156c
Merge blocks starting from the first encrypted
islamaliev Aug 7, 2024
cf4ac1c
Add more options to AES encryption/decryption
islamaliev Aug 7, 2024
e5e5fc2
Add associated data to ECIES
islamaliev Aug 7, 2024
8bd3880
Improve session handling
islamaliev Aug 7, 2024
6ad6061
Merge remote-tracking branch 'upstream/develop' into feat/encryption-…
islamaliev Aug 7, 2024
5e72e9c
Split server into 2 files
islamaliev Aug 7, 2024
7357e9a
Polish
islamaliev Aug 7, 2024
60bbb91
Improve documentation
islamaliev Aug 7, 2024
cdf797c
Polish
islamaliev Aug 7, 2024
bf9d685
Minor improvements
islamaliev Aug 7, 2024
42648bc
Remove unnecessary method
islamaliev Aug 8, 2024
1aafe65
Fixed encryptor tests
islamaliev Aug 8, 2024
5cd8977
Polish
islamaliev Aug 8, 2024
f79bf47
Patch for change detector
islamaliev Aug 8, 2024
f0c3cde
Adjust encryption to work with sec. indexes
islamaliev Aug 8, 2024
20f574c
Pass EncStoreDocKey to encryptor
islamaliev Aug 12, 2024
dcf36e7
Store enc key CID in a block instead of height
islamaliev Aug 13, 2024
c70571d
Merge remote-tracking branch 'upstream/develop' into feat/encryption-…
islamaliev Aug 13, 2024
21c98e7
Add minor PR changes
islamaliev Aug 13, 2024
09fb855
Wait for sync on goroutine if count > 1
islamaliev Aug 13, 2024
f85e9a2
Add docs
islamaliev Aug 13, 2024
a8415bf
Add tests for ECIES
islamaliev Aug 13, 2024
c836968
Add AES tests
islamaliev Aug 13, 2024
fdfa6d4
Polish
islamaliev Aug 13, 2024
25d4373
Add unmarshal tests for block
islamaliev Aug 13, 2024
658a092
Add tests for Cid
islamaliev Aug 13, 2024
168f943
Skip even attempt to index if doc is encrypted
islamaliev Aug 13, 2024
ff277b6
Fix tests
islamaliev Aug 13, 2024
20a1c79
Don't request enc keys if not pending
islamaliev Aug 13, 2024
3a8f09b
Handle AnyOf for doc (not only fields)
islamaliev Aug 13, 2024
53a3e95
Add a test
islamaliev Aug 13, 2024
89ff5f0
Fix lint
islamaliev Aug 13, 2024
16b175d
Fix an issue with overwriting AAD
islamaliev Aug 14, 2024
f115a14
Remove cache from encryptor
islamaliev Aug 14, 2024
0831b01
Make block.GetPrevBlockCids return all heads
islamaliev Aug 14, 2024
af9009b
Moved schemaRoot to session
islamaliev Aug 14, 2024
c8d2d44
Rename Id to ID
islamaliev Aug 16, 2024
bfd89c2
PR polish
islamaliev Aug 16, 2024
c02e5f7
Adjust phony
islamaliev Aug 16, 2024
a01faa1
Fix mistake in AAD
islamaliev Aug 18, 2024
e19cd2b
Remove unnecessary encryptor test
islamaliev Aug 19, 2024
ecd5082
Remove unused exch field of Peer struct
islamaliev Aug 19, 2024
0f561a7
Remove global functions from encryption package
islamaliev Aug 19, 2024
ae716eb
Add more docs
islamaliev Aug 19, 2024
260fa59
Merge remote-tracking branch 'upstream/develop' into feat/encryption-…
islamaliev Sep 1, 2024
714fb3c
Initial KMS implementation
islamaliev Sep 8, 2024
7ff6019
Keep only 1 executeMerge (WIP)
islamaliev Sep 9, 2024
55965e6
Make it work with 2 events
islamaliev Sep 10, 2024
807c252
FIx indexing after decryption
islamaliev Sep 10, 2024
3deb0c3
Change KMS test setup, Rename p2p KMS to pubsub
islamaliev Sep 10, 2024
9dc089b
Strong error types for crypto package
islamaliev Sep 10, 2024
302191c
Use response chan instead of another event
islamaliev Sep 10, 2024
4e35ca0
Polish
islamaliev Sep 10, 2024
88519eb
Remove unused method
islamaliev Sep 10, 2024
c4f8270
Polish
islamaliev Sep 12, 2024
4d97c06
Make encryption key be store in dedicated IPLD block
islamaliev Sep 14, 2024
f84a4d4
Add mocks for encstore
islamaliev Sep 14, 2024
a11d199
Remove unused files
islamaliev Sep 14, 2024
d2fec1f
Fix lint
islamaliev Sep 14, 2024
a9e286f
Add options to ECIES
islamaliev Sep 14, 2024
ab05781
Remove EncStoreKey
islamaliev Sep 14, 2024
bae47d7
Polish
islamaliev Sep 14, 2024
73e7069
Request encBlocks' cids in batches
islamaliev Sep 15, 2024
c7121a4
Minor refactor
islamaliev Sep 15, 2024
0a218ab
Make KMS also wait on ctx.Done()
islamaliev Sep 15, 2024
16d1f8e
Polish
islamaliev Sep 16, 2024
9045ffe
Merge remote-tracking branch 'upstream/develop' into feat/encryption-…
islamaliev Sep 16, 2024
32f2471
Update go mod
islamaliev Sep 16, 2024
b2c1f9d
Lint polish
islamaliev Sep 16, 2024
913f8d7
Merge remote-tracking branch 'upstream/develop' into feat/encryption-…
islamaliev Sep 16, 2024
c2acc3c
Merge remote-tracking branch 'upstream/develop' into feat/encryption-…
islamaliev Sep 17, 2024
ab44314
Add comments
islamaliev Sep 17, 2024
3a72abb
Fix lint
islamaliev Sep 17, 2024
05a4b9a
Merge remote-tracking branch 'upstream/develop' into feat/encryption-…
islamaliev Sep 17, 2024
673bf54
Merge remote-tracking branch 'upstream/develop' into feat/encryption-…
islamaliev Sep 17, 2024
bb1466a
Merge remote-tracking branch 'upstream/develop' into feat/encryption-…
islamaliev Sep 19, 2024
ef9022f
PR fix up
islamaliev Sep 19, 2024
bc335af
Add tests for checking encryption of empty and nil values
islamaliev Sep 19, 2024
8cdb515
Merge remote-tracking branch 'upstream/develop' into feat/encryption-…
islamaliev Sep 19, 2024
90866e1
Merge remote-tracking branch 'upstream/develop' into feat/encryption-…
islamaliev Sep 21, 2024
a492a1d
PR fixup
islamaliev Sep 23, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Polish
islamaliev committed Sep 16, 2024
commit 16d1f8e8f48a58b06dc8792847d4c6a75d696214
2 changes: 1 addition & 1 deletion internal/db/merge.go
Original file line number Diff line number Diff line change
@@ -277,7 +277,7 @@ func (mp *mergeProcessor) tryFetchMissingBlocksAndMerge(ctx context.Context) err
mp.availableEncryptionBlocks[cidlink.Link{Cid: link}] = &encBlock
}

err := mp.mergeComposites(context.Background())
err := mp.mergeComposites(ctx)
if err != nil {
return err
}
5 changes: 5 additions & 0 deletions internal/kms/pubsub.go
Original file line number Diff line number Diff line change
@@ -60,6 +60,11 @@ func (s *pubSubService) GetKeys(ctx context.Context, cids ...cidlink.Link) (*enc
return res, nil
}

// NewPubSubService creates a new instance of the KMS service that is connected to the given PubSubServer,
// event bus and encryption storage.
//
// The service will subscribe to the "encryption" topic on the PubSubServer and to the
// "enc-keys-request" event on the event bus.
func NewPubSubService(
ctx context.Context,
peerID libpeer.ID,
6 changes: 6 additions & 0 deletions internal/kms/service.go
Original file line number Diff line number Diff line change
@@ -26,9 +26,15 @@ var (
type ServiceType string

const (
// PubSubServiceType is the type of KMS that uses PubSub mechanism to exchange keys
// between peers.
PubSubServiceType ServiceType = "pubsub"
)

// Service is interface for key management service (KMS)
type Service interface {
// GetKeys retrieves the encryption blocks containing encryption keys for the given links.
// Blocks are fetched asynchronously, so the method returns an [encryption.Results] object
// that can be used to wait for the results.
GetKeys(ctx context.Context, cids ...cidlink.Link) (*encryption.Results, error)
}
2 changes: 0 additions & 2 deletions internal/merkle/clock/clock.go
Original file line number Diff line number Diff line change
@@ -222,8 +222,6 @@ func encryptBlock(
}

// ProcessBlock merges the delta CRDT and updates the state accordingly.
// If skipMerge is true, it will skip merging and update only the heads.
// If skipHeads is true, it will skip updating the heads.
func (mc *MerkleClock) ProcessBlock(
ctx context.Context,
block *coreblock.Block,