⁂
web3.storage

The filecoin pipeline infra for w3protocol built on SST. w3filecoin is responsible for keeping the state over time from CAR files being ready to aggregate, to an aggregate landing in Filecoin Storage providers.

Getting Started

The repo contains the infra deployment code and the service implementation.

├── stacks      - sst and aws cdk code to deploy all the things
└── api     - lambda & dynamoDB implementations for the filecoin pipeline

To work on this codebase you need:

• Node.js >= v18 (prod env is node v18)
• An AWS account with the AWS CLI configured locally
• Copy .env.tpl to .env.local
• Install the deps with npm i

Deploy dev services to your aws account and start dev console

npm start

See: https://docs.sst.dev for more info on how things get deployed.

Deployment

Deployment is managed by seed.run. PR's are deployed automatically to https://<pr#>.filecoin.web3.storage.

The main branch is deployed to https://staging.filecoin.web3.storage and staging builds are promoted to prod manually via the UI at https://console.seed.run

Local dev

You can use sst to create a custom dev deployment on aws, with a local dev console for debugging.

To do that you need

• An AWS account with the AWS CLI configured locally
• Copy .env.tpl to .env.local

Then run npm dev to deploy dev services to your aws account and start dev console

See: https://docs.sst.dev for more info on how things get deployed.

Environment Variables

Ensure the following variables are set in the env when deploying

AGGREGATOR_HOSTED_ZONE

The root domain to deploy the aggregator API to. e.g aggregator.web3.storage. The value should match a hosted zone configured in route53 that your aws account has access to.

DEAL_TRACKER_API_HOSTED_ZONE

The root domain to deploy the deal tracker API to. e.g tracker.web3.storage. The value should match a hosted zone configured in route53 that your aws account has access to.

DEALER_API_HOSTED_ZONE

The root domain to deploy the dealer API to. e.g dealer.web3.storage. The value should match a hosted zone configured in route53 that your aws account has access to.

AGGREGATOR_DID

DID of the ucanto server running for the Aggregator service. e.g. did:key:abc... Optional: if omitted, a did:key will be derrived from AGGREGATOR_PRIVATE_KEY

DEAL_TRACKER_DID

DID of the ucanto server running for the Deal Tracker service. e.g. did:key:abc... Optional: if omitted, a did:key will be derrived from DEAL_TRACKER_PRIVATE_KEY

DEALER_DID

DID of the ucanto server running a Dealer service. e.g. did:web:dealer.web3.storage.

UCAN_LOG_URL

URL of the UCAN log server. e.g. https://up.web3.storage/ucan

Secrets

Set production secrets in aws SSM via sst secrets. The region must be set to the one you deploy that stage to

# set `PRIVATE_KEY` for prod
$ npx sst secrets set --region us-west-2 --stage prod PRIVATE_KEY "MgCblCY...="

To set a fallback value for staging or an ephmeral PR build use sst secrets set-fallback

# set `PRIVATE_KEY` for any stage in us-east-2
$ npx sst secrets set --fallback --region us-east-2 PRIVATE_KEY "MgCZG7...="

note The fallback value can only be inherited by stages deployed in the same AWS account and region.

Confirm the secret value using sst secrets list

$ npx sst secrets list --region us-east-2
PRIVATE_KEY MgCZG7...= (fallback)

$ npx sst secrets list --region us-west-2 --stage prod
PRIVATE_KEY M...=

AGGREGATOR_PRIVATE_KEY

The base64pad multibase encoded ED25519 keypair used as the signing key for the aggregator-api.

Generated by @ucanto/principal EdSigner via ucan-key

Example: MgCZG7EvaA...1pX9as=

DEALER_PRIVATE_KEY

The base64pad multibase encoded ED25519 keypair used as the signing key for the dealer-api.

Generated by @ucanto/principal EdSigner via ucan-key

Example: MgCZG7EvaA...1pX9as=

DEAL_TRACKER_PRIVATE_KEY

The multibase encoded ED25519 keypair used as the signing key for the deal-tracker-api.

Generated by @ucanto/principal EdSigner via ucan-key

Example: MgCZG7EvaA...1pX9as=

UCAN_LOG_BASIC_AUTH

The HTTP Basic auth token for the UCAN Invocation entrypoint, where UCAN invocations can be stored and proxied to the UCAN Stream.

Example: MgCZG7EvaA...1pX9as=