Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Secure document encryption key exchange #2891

Merged
Merged
Show file tree
Hide file tree
Changes from 52 commits
Commits
Show all changes
101 commits
Select commit Hold shift + click to select a range
8a4b322
Update protobuf
islamaliev Jul 13, 2024
8c81b46
Fix after rebase
islamaliev Jul 17, 2024
b6ef757
Enable very naive key exchange
islamaliev Jul 24, 2024
5d18970
Add protobuf data for key request/response
islamaliev Jul 27, 2024
83cb8da
Encrypt peer-to-peer data exchange
islamaliev Jul 27, 2024
3153090
Merge remote-tracking branch 'upstream/develop' into feat/encryption-…
islamaliev Jul 27, 2024
c2ff5a3
Doc field key exchange
islamaliev Jul 30, 2024
1505ace
Decrypt doc-level and field-level block simultaneously
islamaliev Aug 1, 2024
4782ea5
Add encryption with ECDH
islamaliev Aug 3, 2024
5d83085
Remove unnecessary pub key transit
islamaliev Aug 3, 2024
cf5c559
Implement ECIES
islamaliev Aug 3, 2024
d46b881
Adjustments
islamaliev Aug 3, 2024
4eafe57
Polish
islamaliev Aug 4, 2024
abb3f99
Remove unnecessary peerInfo transit
islamaliev Aug 4, 2024
58bf7ce
Polish
islamaliev Aug 4, 2024
9d89392
Merge remote-tracking branch 'upstream/develop' into feat/encryption-…
islamaliev Aug 4, 2024
a24c9b0
Make all mission enc keys be batched
islamaliev Aug 5, 2024
31e0c37
Load latest available encrypted block from Blockstore instead of fetc…
islamaliev Aug 5, 2024
809e181
Make block store height of where encryption started
islamaliev Aug 7, 2024
a127298
Make failed tests show this 'path: commits[2].links[1].cid' instead o…
islamaliev Aug 7, 2024
f62156c
Merge blocks starting from the first encrypted
islamaliev Aug 7, 2024
cf4ac1c
Add more options to AES encryption/decryption
islamaliev Aug 7, 2024
e5e5fc2
Add associated data to ECIES
islamaliev Aug 7, 2024
8bd3880
Improve session handling
islamaliev Aug 7, 2024
6ad6061
Merge remote-tracking branch 'upstream/develop' into feat/encryption-…
islamaliev Aug 7, 2024
5e72e9c
Split server into 2 files
islamaliev Aug 7, 2024
7357e9a
Polish
islamaliev Aug 7, 2024
60bbb91
Improve documentation
islamaliev Aug 7, 2024
cdf797c
Polish
islamaliev Aug 7, 2024
bf9d685
Minor improvements
islamaliev Aug 7, 2024
42648bc
Remove unnecessary method
islamaliev Aug 8, 2024
1aafe65
Fixed encryptor tests
islamaliev Aug 8, 2024
5cd8977
Polish
islamaliev Aug 8, 2024
f79bf47
Patch for change detector
islamaliev Aug 8, 2024
f0c3cde
Adjust encryption to work with sec. indexes
islamaliev Aug 8, 2024
20f574c
Pass EncStoreDocKey to encryptor
islamaliev Aug 12, 2024
dcf36e7
Store enc key CID in a block instead of height
islamaliev Aug 13, 2024
c70571d
Merge remote-tracking branch 'upstream/develop' into feat/encryption-…
islamaliev Aug 13, 2024
21c98e7
Add minor PR changes
islamaliev Aug 13, 2024
09fb855
Wait for sync on goroutine if count > 1
islamaliev Aug 13, 2024
f85e9a2
Add docs
islamaliev Aug 13, 2024
a8415bf
Add tests for ECIES
islamaliev Aug 13, 2024
c836968
Add AES tests
islamaliev Aug 13, 2024
fdfa6d4
Polish
islamaliev Aug 13, 2024
25d4373
Add unmarshal tests for block
islamaliev Aug 13, 2024
658a092
Add tests for Cid
islamaliev Aug 13, 2024
168f943
Skip even attempt to index if doc is encrypted
islamaliev Aug 13, 2024
ff277b6
Fix tests
islamaliev Aug 13, 2024
20a1c79
Don't request enc keys if not pending
islamaliev Aug 13, 2024
3a8f09b
Handle AnyOf for doc (not only fields)
islamaliev Aug 13, 2024
53a3e95
Add a test
islamaliev Aug 13, 2024
89ff5f0
Fix lint
islamaliev Aug 13, 2024
16b175d
Fix an issue with overwriting AAD
islamaliev Aug 14, 2024
f115a14
Remove cache from encryptor
islamaliev Aug 14, 2024
0831b01
Make block.GetPrevBlockCids return all heads
islamaliev Aug 14, 2024
af9009b
Moved schemaRoot to session
islamaliev Aug 14, 2024
c8d2d44
Rename Id to ID
islamaliev Aug 16, 2024
bfd89c2
PR polish
islamaliev Aug 16, 2024
c02e5f7
Adjust phony
islamaliev Aug 16, 2024
a01faa1
Fix mistake in AAD
islamaliev Aug 18, 2024
e19cd2b
Remove unnecessary encryptor test
islamaliev Aug 19, 2024
ecd5082
Remove unused exch field of Peer struct
islamaliev Aug 19, 2024
0f561a7
Remove global functions from encryption package
islamaliev Aug 19, 2024
ae716eb
Add more docs
islamaliev Aug 19, 2024
260fa59
Merge remote-tracking branch 'upstream/develop' into feat/encryption-…
islamaliev Sep 1, 2024
714fb3c
Initial KMS implementation
islamaliev Sep 8, 2024
7ff6019
Keep only 1 executeMerge (WIP)
islamaliev Sep 9, 2024
55965e6
Make it work with 2 events
islamaliev Sep 10, 2024
807c252
FIx indexing after decryption
islamaliev Sep 10, 2024
3deb0c3
Change KMS test setup, Rename p2p KMS to pubsub
islamaliev Sep 10, 2024
9dc089b
Strong error types for crypto package
islamaliev Sep 10, 2024
302191c
Use response chan instead of another event
islamaliev Sep 10, 2024
4e35ca0
Polish
islamaliev Sep 10, 2024
88519eb
Remove unused method
islamaliev Sep 10, 2024
c4f8270
Polish
islamaliev Sep 12, 2024
4d97c06
Make encryption key be store in dedicated IPLD block
islamaliev Sep 14, 2024
f84a4d4
Add mocks for encstore
islamaliev Sep 14, 2024
a11d199
Remove unused files
islamaliev Sep 14, 2024
d2fec1f
Fix lint
islamaliev Sep 14, 2024
a9e286f
Add options to ECIES
islamaliev Sep 14, 2024
ab05781
Remove EncStoreKey
islamaliev Sep 14, 2024
bae47d7
Polish
islamaliev Sep 14, 2024
73e7069
Request encBlocks' cids in batches
islamaliev Sep 15, 2024
c7121a4
Minor refactor
islamaliev Sep 15, 2024
0a218ab
Make KMS also wait on ctx.Done()
islamaliev Sep 15, 2024
16d1f8e
Polish
islamaliev Sep 16, 2024
9045ffe
Merge remote-tracking branch 'upstream/develop' into feat/encryption-…
islamaliev Sep 16, 2024
32f2471
Update go mod
islamaliev Sep 16, 2024
b2c1f9d
Lint polish
islamaliev Sep 16, 2024
913f8d7
Merge remote-tracking branch 'upstream/develop' into feat/encryption-…
islamaliev Sep 16, 2024
c2acc3c
Merge remote-tracking branch 'upstream/develop' into feat/encryption-…
islamaliev Sep 17, 2024
ab44314
Add comments
islamaliev Sep 17, 2024
3a72abb
Fix lint
islamaliev Sep 17, 2024
05a4b9a
Merge remote-tracking branch 'upstream/develop' into feat/encryption-…
islamaliev Sep 17, 2024
673bf54
Merge remote-tracking branch 'upstream/develop' into feat/encryption-…
islamaliev Sep 17, 2024
bb1466a
Merge remote-tracking branch 'upstream/develop' into feat/encryption-…
islamaliev Sep 19, 2024
ef9022f
PR fix up
islamaliev Sep 19, 2024
bc335af
Add tests for checking encryption of empty and nil values
islamaliev Sep 19, 2024
8cdb515
Merge remote-tracking branch 'upstream/develop' into feat/encryption-…
islamaliev Sep 19, 2024
90866e1
Merge remote-tracking branch 'upstream/develop' into feat/encryption-…
islamaliev Sep 21, 2024
a492a1d
PR fixup
islamaliev Sep 23, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion cli/collection_create.go
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,7 @@
}
ctx := cmd.Context()
if txn != nil {
islamaliev marked this conversation as resolved.
Show resolved Hide resolved
ctx = encryption.ContextWithStore(ctx, txn)
ctx = encryption.ContextWithStore(ctx, txn.Encstore())

Check warning on line 125 in cli/collection_create.go

View check run for this annotation

Codecov / codecov/patch

cli/collection_create.go#L125

Added line #L125 was not covered by tests
islamaliev marked this conversation as resolved.
Show resolved Hide resolved
}
ctx = encryption.SetContextConfigFromParams(ctx, shouldEncryptDoc, encryptFields)
cmd.SetContext(ctx)
Expand Down
5 changes: 5 additions & 0 deletions client/db.go
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,11 @@ type DB interface {
// It sits within the rootstore returned by [Root].
Blockstore() datastore.Blockstore

// Encstore returns the store, that contains all known encryption keys for documents and their fields.
//
// It sits within the rootstore returned by [Root].
Encstore() datastore.DSReaderWriter

// Peerstore returns the peerstore where known host information is stored.
//
// It sits within the rootstore returned by [Root].
Expand Down
47 changes: 47 additions & 0 deletions client/mocks/db.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

95 changes: 95 additions & 0 deletions crypto/aes.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
// Copyright 2024 Democratized Data Foundation
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.

package crypto

import (
"crypto/aes"
"crypto/cipher"
"fmt"
)

// EncryptAES encrypts data using AES-GCM with a provided key and additional data.
// It generates a nonce internally and optionally prepends it to the cipherText.
//
// Parameters:
// - plainText: The data to be encrypted
// - key: The AES encryption key
// - additionalData: Additional authenticated data (AAD) to be used in the encryption
// - prependNonce: If true, the nonce is prepended to the returned cipherText
//
// Returns:
// - cipherText: The encrypted data, with the nonce prepended if prependNonce is true
// - nonce: The generated nonce
// - error: Any error encountered during the encryption process
islamaliev marked this conversation as resolved.
Show resolved Hide resolved
func EncryptAES(plainText, key, additionalData []byte, prependNonce bool) ([]byte, []byte, error) {
block, err := aes.NewCipher(key)
if err != nil {
return nil, nil, err
}

nonce, err := generateNonceFunc()
if err != nil {
return nil, nil, err

Check warning on line 40 in crypto/aes.go

View check run for this annotation

Codecov / codecov/patch

crypto/aes.go#L40

Added line #L40 was not covered by tests
}

aesGCM, err := cipher.NewGCM(block)
if err != nil {
return nil, nil, err

Check warning on line 45 in crypto/aes.go

View check run for this annotation

Codecov / codecov/patch

crypto/aes.go#L45

Added line #L45 was not covered by tests
}

var cipherText []byte
if prependNonce {
cipherText = aesGCM.Seal(nonce, nonce, plainText, additionalData)
} else {
cipherText = aesGCM.Seal(nil, nonce, plainText, additionalData)
}

return cipherText, nonce, nil
}

// DecryptAES decrypts AES-GCM encrypted data with a provided key and additional data.
// If no separate nonce is provided, it assumes the nonce is prepended to the cipherText.
//
// Parameters:
// - nonce: The nonce used for decryption. If empty, it's assumed to be prepended to cipherText
// - cipherText: The data to be decrypted
// - key: The AES decryption key
// - additionalData: Additional authenticated data (AAD) used during encryption
//
// Returns:
// - plainText: The decrypted data
// - error: Any error encountered during the decryption process, including authentication failures
func DecryptAES(nonce, cipherText, key, additionalData []byte) ([]byte, error) {
if len(nonce) == 0 {
if len(cipherText) < AESNonceSize {
return nil, fmt.Errorf("cipherText too short")
}
nonce = cipherText[:AESNonceSize]
cipherText = cipherText[AESNonceSize:]
}

block, err := aes.NewCipher(key)
if err != nil {
return nil, err
}

aesGCM, err := cipher.NewGCM(block)
if err != nil {
return nil, err

Check warning on line 86 in crypto/aes.go

View check run for this annotation

Codecov / codecov/patch

crypto/aes.go#L86

Added line #L86 was not covered by tests
}

plainText, err := aesGCM.Open(nil, nonce, cipherText, additionalData)
if err != nil {
return nil, err
}

return plainText, nil
}
175 changes: 175 additions & 0 deletions crypto/aes_test.go
islamaliev marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
@@ -0,0 +1,175 @@
// Copyright 2024 Democratized Data Foundation
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.

package crypto

import (
"bytes"
"crypto/rand"
"testing"

"github.com/stretchr/testify/require"
)

func TestEncryptAES(t *testing.T) {
validKey := make([]byte, 32) // AES-256
_, err := rand.Read(validKey)
require.NoError(t, err)
validPlaintext := []byte("Hello, World!")
validAAD := []byte("Additional Authenticated Data")

tests := []struct {
name string
plainText []byte
key []byte
additionalData []byte
prependNonce bool
expectError bool
errorContains string
}{
{
name: "Valid encryption with prepended nonce",
plainText: validPlaintext,
key: validKey,
additionalData: validAAD,
prependNonce: true,
expectError: false,
},
{
name: "Valid encryption without prepended nonce",
plainText: validPlaintext,
key: validKey,
additionalData: validAAD,
prependNonce: false,
expectError: false,
},
{
name: "Invalid key size",
plainText: validPlaintext,
key: make([]byte, 31), // Invalid key size
additionalData: validAAD,
prependNonce: true,
expectError: true,
errorContains: "invalid key size",
},
{
name: "Nil plaintext",
plainText: nil,
key: validKey,
additionalData: validAAD,
prependNonce: true,
expectError: false, // AES-GCM can encrypt nil/empty plaintext
},
{
name: "Nil additional data",
plainText: validPlaintext,
key: validKey,
additionalData: nil,
prependNonce: true,
expectError: false, // Nil AAD is valid
},
}

for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
cipherText, nonce, err := EncryptAES(tt.plainText, tt.key, tt.additionalData, tt.prependNonce)

if tt.expectError {
require.Error(t, err)
require.Contains(t, err.Error(), tt.errorContains)
} else {
require.NoError(t, err)
if tt.prependNonce {
require.Greater(t, len(cipherText), len(nonce), "Ciphertext length not greater than nonce length")
} else {
require.Equal(t, AESNonceSize, len(nonce), "Nonce length != AESNonceSize")
}
}
})
}
}

func TestDecryptAES(t *testing.T) {
validKey := make([]byte, 32) // AES-256
_, err := rand.Read(validKey)
require.NoError(t, err)
validPlaintext := []byte("Hello, World!")
validAAD := []byte("Additional Authenticated Data")
validCiphertext, validNonce, _ := EncryptAES(validPlaintext, validKey, validAAD, true)

tests := []struct {
name string
nonce []byte
cipherText []byte
key []byte
additionalData []byte
expectError bool
errorContains string
}{
{
name: "Valid decryption",
nonce: nil, // Should be extracted from cipherText
cipherText: validCiphertext,
key: validKey,
additionalData: validAAD,
expectError: false,
},
{
name: "Invalid key size",
nonce: validNonce,
cipherText: validCiphertext[AESNonceSize:],
key: make([]byte, 31), // Invalid key size
additionalData: validAAD,
expectError: true,
errorContains: "invalid key size",
},
{
name: "Ciphertext too short",
nonce: nil,
cipherText: make([]byte, AESNonceSize-1), // Too short to contain nonce
key: validKey,
additionalData: validAAD,
expectError: true,
errorContains: "cipherText too short",
},
{
name: "Invalid additional data",
nonce: validNonce,
cipherText: validCiphertext[AESNonceSize:],
key: validKey,
additionalData: []byte("Wrong AAD"),
expectError: true,
errorContains: "message authentication failed",
},
{
name: "Tampered ciphertext",
nonce: validNonce,
cipherText: append([]byte{0}, validCiphertext[AESNonceSize+1:]...),
key: validKey,
additionalData: validAAD,
expectError: true,
errorContains: "message authentication failed",
},
}

for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
plainText, err := DecryptAES(tt.nonce, tt.cipherText, tt.key, tt.additionalData)

if tt.expectError {
require.Error(t, err)
require.Contains(t, err.Error(), tt.errorContains)
} else {
require.NoError(t, err)
require.True(t, bytes.Equal(plainText, validPlaintext), "Decrypted plaintext does not match original")
}
})
}
}
26 changes: 26 additions & 0 deletions crypto/cid.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
// Copyright 2024 Democratized Data Foundation
//
// Use of this software is governed by the Business Source License
// included in the file licenses/BSL.txt.
//
// As of the Change Date specified in that file, in accordance with
// the Business Source License, use of this software will be governed
// by the Apache License, Version 2.0, included in the file
// licenses/APL.txt.

package crypto

import (
"github.com/ipfs/go-cid"
"github.com/multiformats/go-multihash"
)

// GenerateCid generates a CID from the given data.
func GenerateCid(data []byte) (cid.Cid, error) {
mh, err := multihash.Sum(data, multihash.SHA2_256, -1)
if err != nil {
return cid.Cid{}, err

Check warning on line 22 in crypto/cid.go

View check run for this annotation

Codecov / codecov/patch

crypto/cid.go#L22

Added line #L22 was not covered by tests
}

return cid.NewCidV1(cid.Raw, mh), nil
}
Loading
Loading