Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(main): eigen client m0 implementation #347

Open
wants to merge 64 commits into
base: eigen-client-extra-features
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
fd913fe
chore: ignore derivative advisory (#3247)
perekopskiy Nov 11, 2024
6ea36d1
feat(api-server): add `yParity` for non-legacy txs (#3246)
itegulov Nov 11, 2024
cce765b
refactor(utils): Recombobulate utils crate (#3237)
slowli Nov 11, 2024
cc4cd8d
ci: migrate release workflows to new reusable workflows (#3198)
otani88 Nov 11, 2024
8552d9c
chore: Refine loadtest configurations (#2649)
encody Nov 11, 2024
8b31a85
test(merkle-tree): Fix RocksDB termination in test (#3249)
slowli Nov 11, 2024
5cea9ec
chore: Add README.md with basic information about Prover Autoscaler (…
yorik Nov 11, 2024
6e3c36e
fix: use_dummy_inclusion_data condition (#3244)
dimazhornyk Nov 12, 2024
8be0c65
ci: migrate docker from tag workflow to new templates (#3250)
otani88 Nov 12, 2024
363b4f0
fix(merkle-tree): Repair stale keys for tree in background (#3200)
slowli Nov 12, 2024
75f7db9
feat(contract-verifier): Support Vyper toolchain for EVM bytecodes (#…
slowli Nov 12, 2024
407bb3d
ci: revert migrate release workflows to new reusable workflows (#3263)
otani88 Nov 12, 2024
5158875
ci: revert migrate docker from tag workflow to new templates #2 (#3264)
otani88 Nov 12, 2024
88a8940
fix: add deploy timestamp asserter command (#3252)
ischasny Nov 12, 2024
64d0a38
feat(eigen-client-m0-implementation): concurrent da dispatcher (#333)
juan518munoz Nov 12, 2024
e92b806
feat(zkstack): Introduce `build` and `wait` subcommands for component…
slowli Nov 12, 2024
f3a2517
feat(vm): add gateway changes to fast vm (#3236)
perekopskiy Nov 12, 2024
f783022
docs(consensus): Restored decentralization instructions (#3254)
brunoffranca Nov 12, 2024
4afbc31
docs(zkstack_cli): update docs to mention foundry zksync (#3134)
manuelmauro Nov 13, 2024
0a88698
chore(consensus): updated EN config templates (#3269)
pompon0 Nov 13, 2024
726203b
feat(consensus): fallback json rpc syncing for consensus (#3211)
pompon0 Nov 13, 2024
54e4b00
fix(vm): Do not require experimental VM config (#3270)
slowli Nov 13, 2024
8620a8e
fix: Fixes build issues related to foundy-zksync (#3272)
brunoffranca Nov 14, 2024
1edcabe
refactor(utils): Move bytecode hashing to `basic_types` (#3258)
slowli Nov 14, 2024
98803ef
ci: fix reusable workflow for build core (#3268)
otani88 Nov 14, 2024
6844651
ci: fix contract verifier workflow (#3283)
otani88 Nov 14, 2024
bc00c4a
feat: Add min_replicas for SimpleScaler, apply_min_to_namespace confi…
yorik Nov 14, 2024
64e77ce
docs: convert core and prover docs to mdbook project and add CI (#3275)
antonbaliasnikov Nov 15, 2024
8703005
ci: add extra docker tags for prover workflows (#3288)
otani88 Nov 15, 2024
5a0d1f0
ci: update deploy-mdbooks action to fix mermaid display issue (#3289)
antonbaliasnikov Nov 15, 2024
2b4fee8
ci: add backward compatible with older docker images tag naming conve…
otani88 Nov 15, 2024
90b537b
chore: Fix linting of js in docs (#3293)
popzxc Nov 15, 2024
20db220
chore(docs): Improve the core book (#3294)
popzxc Nov 15, 2024
c8ded03
docs(consensus): instructions for enabling consensus on the main node…
pompon0 Nov 15, 2024
271033f
fix(zkstack): Fix loading genesis before cloning zksync-era repo (#3271)
matias-gonz Nov 15, 2024
98823f9
feat: ProverJobProcessor & circuit prover (#3287)
EmilLuta Nov 15, 2024
eeb54d3
ci: revert latest tag for prover subsystem (#3295)
otani88 Nov 15, 2024
c02098b
test: Brush up test contracts (#3035)
slowli Nov 15, 2024
21f0d18
ci: migrate docker from tag workflow to new templates (#3284)
otani88 Nov 18, 2024
e084a63
ci: revert migrate docker from tag workflow to new templates (#3298)
otani88 Nov 18, 2024
ca3c0c6
chore(main): release prover 17.1.0 (#3213)
zksync-era-bot Nov 18, 2024
2d3b0b8
fix: Allow protobuf to clone contracts submodule recursively (#3301)
EmilLuta Nov 18, 2024
88357b8
docs: Update prover docs links (#3302)
EmilLuta Nov 18, 2024
3404e2a
ci: Migrate docker from tag workflow to new templates (#3303)
yorik Nov 18, 2024
8b62434
feat: add more metrics for the tee_prover (#3276)
haraldh Nov 18, 2024
7c93c47
fix(tracer): Add error to flat tracer (#3306)
Deniallugo Nov 19, 2024
eee7429
ci: Make prover e2e test mandatory (#3305)
Artemka374 Nov 19, 2024
b26f939
chore(main): release core 25.2.0 (#3228)
zksync-era-bot Nov 19, 2024
4c5af89
feat(eigen-client-m0-implementation): grafana metrics (#334)
juan518munoz Nov 19, 2024
103ae8e
fix sleep millis instead of secs (#348)
juan518munoz Nov 19, 2024
5aee501
ci: fix tags processing in core reusable template (#3310)
otani88 Nov 20, 2024
920eba1
fix: Add prometheus exporter to circut prover (#3311)
EmilLuta Nov 20, 2024
5e3b125
feat(eigen-client-m0-implementation): send blob id to l1 (#332)
juan518munoz Nov 20, 2024
1715766
docs(consensus): Minor fixes to decentralization docs (#3307)
brunoffranca Nov 20, 2024
3e309e0
feat(proof-data-handler): exclude batches without object file in GCS …
pbeza Nov 21, 2024
a10c4ba
feat(contract-verifier): Download compilers from GH automatically (#3…
popzxc Nov 21, 2024
43a3e5c
Merge branch 'eigen-client-extra-features' into eigen-client-m0-with-…
gianbelinche Nov 22, 2024
28c826c
Merge branch 'main' into eigen-client-extra-features-merge-main
gianbelinche Nov 22, 2024
68abc97
Fix cargo lock
gianbelinche Nov 22, 2024
05ee9b6
Merge branch 'eigen-client-extra-features-merge-main' into eigen-clie…
gianbelinche Nov 22, 2024
f3725c6
Fix conflicts
gianbelinche Nov 22, 2024
c47a0cd
clippy & doc
juan518munoz Nov 22, 2024
9416d8d
Merge pull request #353 from lambdaclass/eigen-client-m0-with-extra
gianbelinche Nov 22, 2024
f91bd58
Merge pull request #355 from lambdaclass/eigen-client-m0-extra-merge-…
juan518munoz Nov 22, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
4 changes: 2 additions & 2 deletions .github/release-please/manifest.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"core": "25.1.0",
"prover": "17.0.0",
"core": "25.2.0",
"prover": "17.1.0",
"zkstack_cli": "0.1.2"
}
12 changes: 8 additions & 4 deletions .github/workflows/build-docker-from-tag.yml
Original file line number Diff line number Diff line change
Expand Up @@ -49,14 +49,15 @@ jobs:
build-push-core-images:
name: Build and push image
needs: [ setup ]
uses: ./.github/workflows/build-core-template.yml
uses: ./.github/workflows/new-build-core-template.yml
if: contains(github.ref_name, 'core')
secrets:
DOCKERHUB_USER: ${{ secrets.DOCKERHUB_USER }}
DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
with:
image_tag_suffix: ${{ needs.setup.outputs.image_tag_suffix }}
en_alpha_release: true
action: "push"

build-push-tee-prover-images:
name: Build and push images
Expand All @@ -73,37 +74,40 @@ jobs:
build-push-contract-verifier:
name: Build and push image
needs: [ setup ]
uses: ./.github/workflows/build-contract-verifier-template.yml
uses: ./.github/workflows/new-build-contract-verifier-template.yml
if: contains(github.ref_name, 'contract_verifier')
secrets:
DOCKERHUB_USER: ${{ secrets.DOCKERHUB_USER }}
DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
with:
image_tag_suffix: ${{ needs.setup.outputs.image_tag_suffix }}
action: "push"

build-push-prover-images:
name: Build and push image
needs: [ setup ]
uses: ./.github/workflows/build-prover-template.yml
uses: ./.github/workflows/new-build-prover-template.yml
if: contains(github.ref_name, 'prover')
with:
image_tag_suffix: ${{ needs.setup.outputs.image_tag_suffix }}
ERA_BELLMAN_CUDA_RELEASE: ${{ vars.ERA_BELLMAN_CUDA_RELEASE }}
CUDA_ARCH: "60;70;75;80;89"
action: "push"
secrets:
DOCKERHUB_USER: ${{ secrets.DOCKERHUB_USER }}
DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}

build-push-witness-generator-image-avx512:
name: Build and push image
needs: [ setup ]
uses: ./.github/workflows/build-witness-generator-template.yml
uses: ./.github/workflows/new-build-witness-generator-template.yml
if: contains(github.ref_name, 'prover')
with:
image_tag_suffix: ${{ needs.setup.outputs.image_tag_suffix }}-avx512
ERA_BELLMAN_CUDA_RELEASE: ${{ vars.ERA_BELLMAN_CUDA_RELEASE }}
CUDA_ARCH: "60;70;75;80;89"
WITNESS_GENERATOR_RUST_FLAGS: "-Ctarget_feature=+avx512bw,+avx512cd,+avx512dq,+avx512f,+avx512vl"
action: "push"
secrets:
DOCKERHUB_USER: ${{ secrets.DOCKERHUB_USER }}
DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
Expand Down
42 changes: 31 additions & 11 deletions .github/workflows/ci-core-reusable.yml
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ jobs:
- name: Init
run: |
ci_run run_retried rustup show

- name: Install zkstack
run: |
ci_run ./zkstack_cli/zkstackup/install -g --path ./zkstack_cli/zkstackup/zkstackup
Expand Down Expand Up @@ -147,10 +147,10 @@ jobs:
--base-token-price-denominator 1 \
--set-as-default false \
--ignore-prerequisites \
--legacy-bridge
--legacy-bridge \
--evm-emulator false

ci_run zkstack ecosystem init --dev --verbose
ci_run zkstack dev contracts --test-contracts

# `sleep 60` because we need to wait until server added all the tokens
- name: Run server
Expand Down Expand Up @@ -201,7 +201,7 @@ jobs:
run: |
ci_run ./zkstack_cli/zkstackup/install -g --path ./zkstack_cli/zkstackup/zkstackup || true
ci_run zkstackup -g --local

- name: Create log directories
run: |
SERVER_LOGS_DIR=logs/server
Expand Down Expand Up @@ -262,7 +262,8 @@ jobs:
--base-token-price-nominator 1 \
--base-token-price-denominator 1 \
--set-as-default false \
--ignore-prerequisites
--ignore-prerequisites \
--evm-emulator false

ci_run zkstack chain init \
--deploy-paymaster \
Expand All @@ -283,7 +284,8 @@ jobs:
--base-token-price-nominator 314 \
--base-token-price-denominator 1000 \
--set-as-default false \
--ignore-prerequisites
--ignore-prerequisites \
--evm-emulator false

ci_run zkstack chain init \
--deploy-paymaster \
Expand All @@ -304,7 +306,8 @@ jobs:
--base-token-price-nominator 1 \
--base-token-price-denominator 1 \
--set-as-default false \
--ignore-prerequisites
--ignore-prerequisites \
--evm-emulator false

ci_run zkstack chain build-transactions --chain offline_chain --l1-rpc-url http://127.0.0.1:8545

Expand Down Expand Up @@ -339,7 +342,8 @@ jobs:
--base-token-price-nominator 314 \
--base-token-price-denominator 1000 \
--set-as-default false \
--ignore-prerequisites
--ignore-prerequisites \
--evm-emulator false

ci_run zkstack chain init \
--deploy-paymaster \
Expand All @@ -357,10 +361,17 @@ jobs:
run: |
ci_run zkstack dev test build

- name: Build tested binaries
run: |
ci_run zkstack server build
ci_run zkstack external-node build
ci_run zkstack contract-verifier build

- name: Initialize Contract verifier
run: |
ci_run zkstack contract-verifier init --zksolc-version=v1.5.3 --zkvyper-version=v1.5.4 --solc-version=0.8.26 --vyper-version=v0.3.10 --era-vm-solc-version=0.8.26-1.0.1 --only --chain era
ci_run zkstack contract-verifier run --chain era &> ${{ env.SERVER_LOGS_DIR }}/contract-verifier-rollup.log &
ci_run zkstack contract-verifier wait --chain era --verbose

- name: Run servers
run: |
Expand All @@ -375,10 +386,14 @@ jobs:
--components=api,tree,eth,state_keeper,housekeeper,commitment_generator,vm_runner_protective_reads,vm_runner_bwip,vm_playground,da_dispatcher,consensus \
&> ${{ env.SERVER_LOGS_DIR }}/consensus.log &

ci_run sleep 5
ci_run zkstack server wait --ignore-prerequisites --verbose --chain era
ci_run zkstack server wait --ignore-prerequisites --verbose --chain validium
ci_run zkstack server wait --ignore-prerequisites --verbose --chain custom_token
ci_run zkstack server wait --ignore-prerequisites --verbose --chain consensus

- name: Setup attester committee for the consensus chain
- name: Set up attester committee for the consensus chain
run: |
ci_run zkstack consensus wait-for-registry --ignore-prerequisites --verbose --chain consensus
ci_run zkstack consensus set-attester-committee --chain consensus --from-genesis &> ${{ env.INTEGRATION_TESTS_LOGS_DIR }}/consensus.log

- name: Run integration tests
Expand Down Expand Up @@ -411,12 +426,17 @@ jobs:
run: |
ci_run ./bin/run_on_all_chains.sh "zkstack dev test recovery --no-deps --no-kill --ignore-prerequisites --verbose" ${{ env.CHAINS }} ${{ env.INTEGRATION_TESTS_LOGS_DIR }}

- name: Run external node server
- name: Run external nodes
run: |
ci_run zkstack external-node run --ignore-prerequisites --chain era &> ${{ env.EXTERNAL_NODE_LOGS_DIR }}/rollup.log &
ci_run zkstack external-node run --ignore-prerequisites --chain validium &> ${{ env.EXTERNAL_NODE_LOGS_DIR }}/validium.log &
ci_run zkstack external-node run --ignore-prerequisites --chain custom_token &> ${{ env.EXTERNAL_NODE_LOGS_DIR }}/custom_token.log &
ci_run zkstack external-node run --ignore-prerequisites --chain consensus --enable-consensus &> ${{ env.EXTERNAL_NODE_LOGS_DIR }}/consensus.log &

ci_run zkstack external-node wait --ignore-prerequisites --verbose --chain era
ci_run zkstack external-node wait --ignore-prerequisites --verbose --chain validium
ci_run zkstack external-node wait --ignore-prerequisites --verbose --chain custom_token
ci_run zkstack external-node wait --ignore-prerequisites --verbose --chain consensus

- name: Run integration tests en
run: |
Expand Down
5 changes: 3 additions & 2 deletions .github/workflows/ci-prover-e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,8 @@ jobs:
--base-token-price-nominator 1 \
--base-token-price-denominator 1 \
--set-as-default true \
--ignore-prerequisites
--ignore-prerequisites \
--evm-emulator false

ci_run zkstack ecosystem init --dev --verbose
ci_run zkstack prover init --dev --verbose
Expand Down Expand Up @@ -86,7 +87,7 @@ jobs:
ci_run zkstack prover run --component=witness-generator --round=all-rounds --docker=false &>prover_logs/witness-generator.log &
- name: Run Circuit Prover
run: |
ci_run zkstack prover run --component=circuit-prover --witness-vector-generator-count=10 --docker=false &>prover_logs/circuit_prover.log &
ci_run zkstack prover run --component=circuit-prover -l=23 -h=3 --docker=false &>prover_logs/circuit_prover.log &
- name: Wait for prover jobs to finish
env:
DATABASE_URL: postgres://postgres:notsecurepassword@localhost:5432/zksync_prover_localhost_proving_chain
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,6 @@ jobs:
outputs:
core: ${{ steps.changed-files.outputs.core_any_changed }}
prover: ${{ steps.changed-files.outputs.prover_any_changed }}
zkstack_cli: ${{ steps.changed-files.outputs.zkstack_cli_any_changed }}
docs: ${{ steps.changed-files.outputs.docs_any_changed }}
all: ${{ steps.changed-files.outputs.all_any_changed }}
steps:
Expand Down Expand Up @@ -178,6 +177,7 @@ jobs:
image_tag_suffix: ${{ needs.setup.outputs.image_tag_suffix }}-avx512
action: "build"
WITNESS_GENERATOR_RUST_FLAGS: "-Ctarget_feature=+avx512bw,+avx512cd,+avx512dq,+avx512f,+avx512vl"
ERA_BELLMAN_CUDA_RELEASE: ${{ vars.ERA_BELLMAN_CUDA_RELEASE }}
secrets:
DOCKERHUB_USER: ${{ secrets.DOCKERHUB_USER }}
DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
Expand All @@ -186,7 +186,7 @@ jobs:
name: Github Status Check
runs-on: ubuntu-latest
if: always() && !cancelled()
needs: [ ci-for-core-lint, ci-for-common, ci-for-core, ci-for-prover, ci-for-docs, build-core-images, build-contract-verifier, build-prover-images ]
needs: [ ci-for-core-lint, ci-for-common, ci-for-core, ci-for-prover, ci-for-docs, build-core-images, build-contract-verifier, build-prover-images, e2e-for-prover ]
steps:
- name: Status
run: |
Expand Down
67 changes: 67 additions & 0 deletions .github/workflows/deploy-core-docs.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
name: Deploy core docs

on:
push:
branches:
- "main"
tags:
- "core-v*.*.*"
paths:
- 'docs/**'
- '.github/workflows/deploy-core-docs.yml'
pull_request:
paths:
- 'docs/**'
- '.github/workflows/deploy-core-docs.yml'
workflow_dispatch:
inputs:
ref:
description: "Branch, tag or commit to deploy the core docs. If empty, use the ref that triggered the workflow."
required: false
default: ""
version:
type: string
description: "Version of the documentation to deploy"
required: false
default: "latest"

# Allow only one concurrent deployment, skipping runs queued between the run in-progress and latest queued.
# However, do NOT cancel in-progress runs as we want to allow these production deployments to complete.
concurrency:
group: "pages"
cancel-in-progress: false

jobs:

deploy-core-docs:
runs-on: ubuntu-latest
permissions:
contents: write
env:
DOCS_DIR: 'docs'
PROJECT: 'core'
ENABLE_TESTS: false
steps:
- name: Checkout
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4
with:
ref: ${{ inputs.ref || '' }}

- name: Extract version from tag
if: startsWith(github.ref, 'refs/tags/')
id: extract-version
shell: 'bash -ex {0}'
run: |
TAG="${{ github.ref_name }}"
VERSION="${TAG#*-}"
echo "version=${VERSION}" >> "${GITHUB_OUTPUT}"

- name: Deploy core docs
uses: matter-labs/deploy-mdbooks@73f638643d1be948d1002fe5433747f4a3e37a29 # v1
with:
version: ${{ inputs.version || steps.extract-version.outputs.version || github.ref_name }}
docs-dir: ${{ env.DOCS_DIR }}
github-token: ${{ secrets.GITHUB_TOKEN }}
enable-tests: ${{ env.ENABLE_TESTS }}
project: ${{ env.PROJECT }}
deploy: ${{ github.event_name != 'pull_request' }}
67 changes: 67 additions & 0 deletions .github/workflows/deploy-prover-docs.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
name: Deploy prover docs

on:
push:
branches:
- "main"
tags:
- "prover-v*.*.*"
paths:
- 'prover/docs/**'
- '.github/workflows/deploy-prover-docs.yml'
pull_request:
paths:
- 'prover/docs/**'
- '.github/workflows/deploy-prover-docs.yml'
workflow_dispatch:
inputs:
ref:
description: "Branch, tag or commit to deploy the prover docs. If empty, use the ref that triggered the workflow."
required: false
default: ""
version:
type: string
description: "Version of the documentation to deploy"
required: false
default: "latest"

# Allow only one concurrent deployment, skipping runs queued between the run in-progress and latest queued.
# However, do NOT cancel in-progress runs as we want to allow these production deployments to complete.
concurrency:
group: "pages"
cancel-in-progress: false

jobs:

deploy-prover-docs:
runs-on: ubuntu-latest
permissions:
contents: write
env:
DOCS_DIR: 'prover/docs'
PROJECT: 'prover'
ENABLE_TESTS: false
steps:
- name: Checkout
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4
with:
ref: ${{ inputs.ref || '' }}

- name: Extract version from tag
if: startsWith(github.ref, 'refs/tags/')
id: extract-version
shell: 'bash -ex {0}'
run: |
TAG="${{ github.ref_name }}"
VERSION="${TAG#*-}"
echo "version=${VERSION}" >> "${GITHUB_OUTPUT}"

- name: Deploy prover docs
uses: matter-labs/deploy-mdbooks@73f638643d1be948d1002fe5433747f4a3e37a29 # v1
with:
version: ${{ inputs.version || steps.extract-version.outputs.version || github.ref_name }}
docs-dir: ${{ env.DOCS_DIR }}
github-token: ${{ secrets.GITHUB_TOKEN }}
enable-tests: ${{ env.ENABLE_TESTS }}
project: ${{ env.PROJECT }}
deploy: ${{ github.event_name != 'pull_request' }}
Loading
Loading