Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Implement SourceHub ACP #2657

Merged
merged 28 commits into from
Jul 16, 2024
Merged

feat: Implement SourceHub ACP #2657

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
03ff346
Boost package timeout
AndrewSisley May 23, 2024
1c95871
Implement SourceHub ACP
AndrewSisley May 21, 2024
8a21968
Update Docs
AndrewSisley Jul 10, 2024
77f8b1b
Add source hub CI job
AndrewSisley May 30, 2024
5ec5678
Add SourceHub make:test command
AndrewSisley May 30, 2024
f55340f
PR FIXUP - Add token fields to readme
AndrewSisley Jul 11, 2024
bb229e9
PR FIXUP - Document new config params
AndrewSisley Jul 11, 2024
318ec88
PR FIXUP - Handle windows filepaths
AndrewSisley Jul 11, 2024
6a220a9
PR FIXUP - Pass around TxSigner interface instead of keyring
AndrewSisley Jul 11, 2024
4eb9fe3
PR FIXUP - Remove dead policy mapper code
AndrewSisley Jul 11, 2024
07d116a
PR FIXUP - Make node acp type string
AndrewSisley Jul 11, 2024
f7f5ff0
PR FIXUP - Doc no-op
AndrewSisley Jul 15, 2024
116347e
PR FIXUP - Rename BearerTokenSignatureScheme
AndrewSisley Jul 15, 2024
d0ef651
PR FIXUP - Remove BearerTokenSignatureScheme mutability
AndrewSisley Jul 15, 2024
3d084b2
PR FIXUP - Document BearerTokenSignatureScheme
AndrewSisley Jul 15, 2024
c395e8d
PR FIXUP - Document Identity BearerToken
AndrewSisley Jul 15, 2024
0c548a1
PR FIXUP remove 'module' from doc
AndrewSisley Jul 15, 2024
86b6eb7
PR FIXUP - Document FromToken
AndrewSisley Jul 15, 2024
99c7d79
PR FIXUP - Document path prop
AndrewSisley Jul 15, 2024
f0f3c88
PR FIXUP - Alias soucrhub sdk import
AndrewSisley Jul 15, 2024
4ccfeaa
PR FIXUP - Rename mapping funcs
AndrewSisley Jul 15, 2024
59901aa
PR FIXUP - Document NewWrapper
AndrewSisley Jul 15, 2024
da3c973
PR FIXUP - Fix typo
AndrewSisley Jul 15, 2024
f5c2726
PR FIXUP - Fix typo
AndrewSisley Jul 15, 2024
7b355f8
PR FIXUP - Doc local-acp only limitation
AndrewSisley Jul 15, 2024
97bd37a
PR FIXUP - Dont generate bearer token unless required
AndrewSisley Jul 15, 2024
994aec1
PR FIXUP - Boost change detector initial subtest speed
AndrewSisley Jul 16, 2024
fdd5a25
PR FIXUP - Bump checkout action version
AndrewSisley Jul 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions .github/workflows/test-and-upload-coverage.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,31 +35,57 @@ jobs:
database-type: [badger-file, badger-memory]
mutation-type: [gql, collection-named, collection-save]
lens-type: [wasm-time]
acp-type: [local]
database-encryption: [false]
include:
- os: ubuntu-latest
client-type: go
database-type: badger-memory
mutation-type: collection-save
lens-type: wasm-time
acp-type: local
database-encryption: true
- os: ubuntu-latest
client-type: go
database-type: badger-memory
mutation-type: collection-save
lens-type: wazero
acp-type: local
database-encryption: false
- os: ubuntu-latest
client-type: go
database-type: badger-memory
mutation-type: collection-save
lens-type: wasmer
acp-type: local
database-encryption: false
- os: ubuntu-latest
client-type: go
database-type: badger-memory
mutation-type: collection-save
lens-type: wasm-time
acp-type: source-hub
database-encryption: false
- os: ubuntu-latest
client-type: http
database-type: badger-memory
mutation-type: collection-save
lens-type: wasm-time
acp-type: source-hub
database-encryption: false
- os: ubuntu-latest
client-type: cli
database-type: badger-memory
mutation-type: collection-save
lens-type: wasm-time
acp-type: source-hub
Comment on lines +60 to +81
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: I think at this point we might as well just make another job in this same workflow file called run-sourcehub-tests with it's own strategy only running the three clients (client-type: [go, http, cli]). Will the extra if: ${{ matrix.acp-type == 'source-hub' }} conditions from this strategy. Then under the needs: ... of the upload-coverage job you can add another name run-sourcehub-tests

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do prefer the current setup. Everything works in the same way and is declared in the same location. It is getting a bit lengthy, but is better than the alternative atm.

database-encryption: false
- os: macos-latest
client-type: go
database-type: badger-memory
mutation-type: collection-save
lens-type: wasm-time
acp-type: local
database-encryption: false
## TODO: https://github.com/sourcenetwork/defradb/issues/2080
## Uncomment the lines below to Re-enable the windows build once this todo is resolved.
Expand All @@ -68,6 +94,7 @@ jobs:
## database-type: badger-memory
## mutation-type: collection-save
## lens-type: wasm-time
## acp-type: local
## database-encryption: false

runs-on: ${{ matrix.os }}
Expand All @@ -87,6 +114,7 @@ jobs:
DEFRA_BADGER_ENCRYPTION: ${{ matrix.database-encryption }}
DEFRA_MUTATION_TYPE: ${{ matrix.mutation-type }}
DEFRA_LENS_TYPE: ${{ matrix.lens-type }}
DEFRA_ACP_TYPE: ${{ matrix.acp-type }}

steps:
- name: Checkout code into the directory
Expand Down Expand Up @@ -143,6 +171,20 @@ jobs:
make deps:modules
make deps:test

# We have to checkout the source-hub repo and install it ourselves because it
# contains replace commands in its go.mod file.
- name: Checkout sourcehub code into the directory
if: ${{ matrix.acp-type == 'source-hub' }}
uses: actions/checkout@v4
with:
repository: sourcenetwork/sourcehub
path: _sourceHub

- name: Install SourceHub CLI
if: ${{ matrix.acp-type == 'source-hub' }}
working-directory: _sourceHub
run: make install

- name: Run integration tests
run: make test:coverage

Expand All @@ -158,6 +200,7 @@ jobs:
_${{ matrix.database-type }}\
_${{ matrix.mutation-type }}\
_${{ matrix.lens-type }}\
_${{ matrix.acp-type }}\
_${{ matrix.database-encryption }}\
"
path: coverage.txt
Expand Down
1 change: 1 addition & 0 deletions CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ The following tools are required in order to build and run the tests within this

- [Go](https://go.dev/doc/install)
- Cargo/rustc, typically installed via [rustup](https://www.rust-lang.org/tools/install)
- [SourceHub](https://github.com/sourcenetwork/sourcehub), installed via `make install`

## Documentation
The overall project documentation can be found at [docs.source.network](https://docs.source.network), and its source at [github.com/sourcenetwork/docs.source.network](https://github.com/sourcenetwork/docs.source.network).
Expand Down
6 changes: 5 additions & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ ifdef BUILD_TAGS
BUILD_FLAGS+=-tags $(BUILD_TAGS)
endif

TEST_FLAGS=-race -shuffle=on -timeout 5m
TEST_FLAGS=-race -shuffle=on -timeout 10m

COVERAGE_DIRECTORY=$(PWD)/coverage
COVERAGE_FILE=coverage.txt
Expand Down Expand Up @@ -256,6 +256,10 @@ test\:gql-mutations:
test\:col-named-mutations:
DEFRA_MUTATION_TYPE=collection-named DEFRA_BADGER_MEMORY=true gotestsum --format pkgname -- $(DEFAULT_TEST_DIRECTORIES)

.PHONY: test\:source-hub
test\:source-hub:
DEFRA_ACP_TYPE=source-hub gotestsum --format pkgname -- $(DEFAULT_TEST_DIRECTORIES)

.PHONY: test\:go
test\:go:
go test $(DEFAULT_TEST_DIRECTORIES) $(TEST_FLAGS)
Expand Down
10 changes: 8 additions & 2 deletions acp/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -436,8 +436,14 @@ To perform authenticated operations you will need to build and sign a JWT token

- `sub` public key of the identity
- `aud` host name of the defradb api

> The `exp` and `nbf` fields should also be set to short-lived durations.
- The `exp` and `nbf` fields should also be set to short-lived durations.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: Perhaps a better separation of local and sourcehub acp specific options as they might use them interchangeably or be confusing if an option is needed for xyz acp type.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are no local only fields. And the sourcehub only ones are very clearly documented as such

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe a sourcehub acp dedicated section somewhere in this markdown file guiding users how to setup sourcehub acp and all the bits they would need, and how to get them?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do think that should be covered by SourceHub documentation. We have to bear in mind that SourceHub is a separate service/process too, and it's version is not tied to Defra releases. By duplicating documentation in the Defra repo it will likely be incorrect at various times as SourceHub updates.


Additionally, if using SourceHub ACP, the following must be set:
- `iss` should be set to the user's DID, e.g. `"did:key:z6MkkHsQbp3tXECqmUJoCJwyuxSKn1BDF1RHzwDGg9tHbXKw"`
- `iat` should be set to the current unix timestamp
- `authorized_account` should be set to the SourceHub address of the account signing SourceHub transactions on your
behalf - WARNING - this will currently enable this account to make any SourceHub as your user for the lifetime of the
token, so please only set this if you fully trust the node/account.

The JWT must be signed with the `secp256k1` private key of the identity you wish to perform actions as.

Expand Down
7 changes: 4 additions & 3 deletions acp/acp.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,10 @@ package acp
import (
"context"

"github.com/sourcenetwork/corelog"
"github.com/sourcenetwork/immutable"

"github.com/sourcenetwork/corelog"
"github.com/sourcenetwork/defradb/acp/identity"
)

var (
Expand Down Expand Up @@ -47,7 +48,7 @@ type ACP interface {
// otherwise returns error.
//
// A policy can not be added without a creator identity (sourcehub address).
AddPolicy(ctx context.Context, creatorID string, policy string) (string, error)
AddPolicy(ctx context.Context, creator identity.Identity, policy string) (string, error)

// ValidateResourceExistsOnValidDPI performs DPI validation of the resource (matching resource name)
// that is on the policy (matching policyID), returns an error upon validation failure.
Expand All @@ -68,7 +69,7 @@ type ACP interface {
// - actorID here is the identity of the actor registering the document object.
RegisterDocObject(
ctx context.Context,
actorID string,
indentity identity.Identity,
policyID string,
resourceName string,
docID string,
Expand Down
14 changes: 8 additions & 6 deletions acp/acp_local.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ import (
"github.com/sourcenetwork/acp_core/pkg/runtime"
"github.com/sourcenetwork/acp_core/pkg/types"
"github.com/sourcenetwork/immutable"

"github.com/sourcenetwork/defradb/acp/identity"
)

const localACPStoreName = "local_acp"
Expand Down Expand Up @@ -105,14 +107,14 @@ func (l *ACPLocal) Close() error {

func (l *ACPLocal) AddPolicy(
ctx context.Context,
creatorID string,
creator identity.Identity,
policy string,
marshalType policyMarshalType,
creationTime *protoTypes.Timestamp,
) (string, error) {
principal, err := auth.NewDIDPrincipal(creatorID)
principal, err := auth.NewDIDPrincipal(creator.DID)
if err != nil {
return "", newErrInvalidActorID(err, creatorID)
return "", newErrInvalidActorID(err, creator.DID)
}
ctx = auth.InjectPrincipal(ctx, principal)

Expand Down Expand Up @@ -152,15 +154,15 @@ func (l *ACPLocal) Policy(

func (l *ACPLocal) RegisterObject(
ctx context.Context,
actorID string,
identity identity.Identity,
policyID string,
resourceName string,
objectID string,
creationTime *protoTypes.Timestamp,
) (RegistrationResult, error) {
principal, err := auth.NewDIDPrincipal(actorID)
principal, err := auth.NewDIDPrincipal(identity.DID)
if err != nil {
return RegistrationResult_NoOp, newErrInvalidActorID(err, actorID)
return RegistrationResult_NoOp, newErrInvalidActorID(err, identity.DID)
}

ctx = auth.InjectPrincipal(ctx, principal)
Expand Down
34 changes: 21 additions & 13 deletions acp/acp_local_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,19 @@ import (
"testing"

"github.com/stretchr/testify/require"

"github.com/sourcenetwork/defradb/acp/identity"
)

var identity1 = "did:key:z7r8os2G88XXBNBTLj3kFR5rzUJ4VAesbX7PgsA68ak9B5RYcXF5EZEmjRzzinZndPSSwujXb4XKHG6vmKEFG6ZfsfcQn"
var identity2 = "did:key:z7r8ooUiNXK8TT8Xjg1EWStR2ZdfxbzVfvGWbA2FjmzcnmDxz71QkP1Er8PP3zyLZpBLVgaXbZPGJPS4ppXJDPRcqrx4F"
var invalidIdentity = "did:something"
var identity1 = identity.Identity{
DID: "did:key:z7r8os2G88XXBNBTLj3kFR5rzUJ4VAesbX7PgsA68ak9B5RYcXF5EZEmjRzzinZndPSSwujXb4XKHG6vmKEFG6ZfsfcQn",
}
var identity2 = identity.Identity{
DID: "did:key:z7r8ooUiNXK8TT8Xjg1EWStR2ZdfxbzVfvGWbA2FjmzcnmDxz71QkP1Er8PP3zyLZpBLVgaXbZPGJPS4ppXJDPRcqrx4F",
}
var invalidIdentity = identity.Identity{
DID: "did:something",
}

var validPolicyID string = "d59f91ba65fe142d35fc7df34482eafc7e99fed7c144961ba32c4664634e61b7"
var validPolicy string = `
Expand Down Expand Up @@ -478,7 +486,7 @@ func Test_LocalACP_InMemory_CheckDocAccess_TrueIfHaveAccessFalseIfNotErrorOtherw
hasAccess, errCheckDocAccess := localACP.CheckDocAccess(
ctx,
ReadPermission,
identity1,
identity1.DID,
validPolicyID,
"",
"",
Expand All @@ -491,7 +499,7 @@ func Test_LocalACP_InMemory_CheckDocAccess_TrueIfHaveAccessFalseIfNotErrorOtherw
hasAccess, errCheckDocAccess = localACP.CheckDocAccess(
ctx,
ReadPermission,
identity1,
identity1.DID,
validPolicyID,
"users",
"documentID_XYZ",
Expand All @@ -513,7 +521,7 @@ func Test_LocalACP_InMemory_CheckDocAccess_TrueIfHaveAccessFalseIfNotErrorOtherw
hasAccess, errCheckDocAccess = localACP.CheckDocAccess(
ctx,
ReadPermission,
identity1,
identity1.DID,
validPolicyID,
"users",
"documentID_XYZ",
Expand All @@ -525,7 +533,7 @@ func Test_LocalACP_InMemory_CheckDocAccess_TrueIfHaveAccessFalseIfNotErrorOtherw
hasAccess, errCheckDocAccess = localACP.CheckDocAccess(
ctx,
ReadPermission,
identity2,
identity2.DID,
validPolicyID,
"users",
"documentID_XYZ",
Expand Down Expand Up @@ -564,7 +572,7 @@ func Test_LocalACP_PersistentMemory_CheckDocAccess_TrueIfHaveAccessFalseIfNotErr
hasAccess, errCheckDocAccess := localACP.CheckDocAccess(
ctx,
ReadPermission,
identity1,
identity1.DID,
validPolicyID,
"",
"",
Expand All @@ -577,7 +585,7 @@ func Test_LocalACP_PersistentMemory_CheckDocAccess_TrueIfHaveAccessFalseIfNotErr
hasAccess, errCheckDocAccess = localACP.CheckDocAccess(
ctx,
ReadPermission,
identity1,
identity1.DID,
validPolicyID,
"users",
"documentID_XYZ",
Expand All @@ -599,7 +607,7 @@ func Test_LocalACP_PersistentMemory_CheckDocAccess_TrueIfHaveAccessFalseIfNotErr
hasAccess, errCheckDocAccess = localACP.CheckDocAccess(
ctx,
ReadPermission,
identity1,
identity1.DID,
validPolicyID,
"users",
"documentID_XYZ",
Expand All @@ -611,7 +619,7 @@ func Test_LocalACP_PersistentMemory_CheckDocAccess_TrueIfHaveAccessFalseIfNotErr
hasAccess, errCheckDocAccess = localACP.CheckDocAccess(
ctx,
ReadPermission,
identity2,
identity2.DID,
validPolicyID,
"users",
"documentID_XYZ",
Expand All @@ -631,7 +639,7 @@ func Test_LocalACP_PersistentMemory_CheckDocAccess_TrueIfHaveAccessFalseIfNotErr
hasAccess, errCheckDocAccess = localACP.CheckDocAccess(
ctx,
ReadPermission,
identity1,
identity1.DID,
validPolicyID,
"users",
"documentID_XYZ",
Expand All @@ -643,7 +651,7 @@ func Test_LocalACP_PersistentMemory_CheckDocAccess_TrueIfHaveAccessFalseIfNotErr
hasAccess, errCheckDocAccess = localACP.CheckDocAccess(
ctx,
ReadPermission,
identity2,
identity2.DID,
validPolicyID,
"users",
"documentID_XYZ",
Expand Down
Loading
Loading