feat: batch trees (#1378)

* fix: add light prover client batch update and append tests

added bloomfilter

batch e2e test failing because filter change doesn't work yet

debugging

batched Merkle tree e2e test works with output queue

fix: e2e test work with input and output queues

chore: rename wipe bloomfilter -> wipe_batch

refactored input queue into mt account

stash

test: rnd account init works

chore: replace is_inserted with state

chore: moved offset to bounded vec, added bounded vec init and deserialize

fixed bounded vec imports

refactor: partial batch zkp updates, flexible sized root history array with seq

implemented batch prove and verification in rust tests untested

stash: batched updates work with simulate transactions test

add comment

stash before account *mut

onchain account init works

test: account compression program batch updates work

feat: add flexbile nullifier to circuit

feat: implement nullifier derivation from tx hash

test: batch

test: bounded vec

cleanup

cleanup

cleanup

feat: add flexible nullifier to update circuit

fix: e2e test

chore: add full asserts to simulate tx test

test: root overwrite test

cleanup

* refactor: update health check timeout in spawn_prover function, rewrite prove_batch_append test

Change CI runner to buildjet-8vcpu-ubuntu-2204 for rust tests

run rust.yml on buildjet-16vcpu-ubuntu-2204

debug rust.yml workflow

Add step to download proving keys for big batches

Remove redundant setup steps and update test configurations

Update ProverConfig to use test-specific circuits

cleaning up after rebase to main

* fix: batch append public input computation (#1330)

Issue:
the batch append zkp public input LeavesHashchainHash is computed
dynamically from the values array during the batched update.
If the values array changes in between proof generation and transaction
execution the batched append update fails.
-> Users can make batched append update transactions fail.

Solution:
compute LeavesHashchainHash incrementally when the output value is
inserted into the value array (the same way we compute it with
nullifiers for input queue insertions).

Changes:
1. switch to updated circuits
2. batch.rs
- add start_index field to derive the leaf_index of values from a value
array
- introduce store_and_hash_value to store an output queue value and add
it to the LeavesHashchainHash
3. batched_queue.rs
- init_queue initialize start_index during account initialization
- use store_and_hash_value instead of store_value when inserting into
output queues
- update batch.start_index when wiping a batch
4. batched_merkle_tree.rs
- remove subtree field (it is not used in new batch append circuit)
- initialize root_history with the zero_value_root
- add hashchain store for leaves_hashchain in
ZeroCopyBatchedMerkleTreeAccount
- add leaf_index to nullifier hash H(value, leaf_index, tx_hash)

Co-authored-by: ananas-block <[email protected]>

* feat: integrate batched state merkle trees into programs (#1335)

* fix: rebase

* feat: rollover state trees (#1371)

* feat: rollover batch state trees

* test: batch state tree rollover

* feat: batch address mt (#1365)

* fix: stop zeroing out commitments proven by inclusion

* feat: batch address tree

feat: forester clears bloom filter

feat: insert zkp batch

feat: read only address

* test: add tests and format

* add readonly account tests

* start refactor read only accounts

* feat: read only accounts

* fix: proof by index check in system program

* refactor: tx hash

* perf: unify circuit public inputs

Issue:
- public inputs consume too many CU onchain

Impacted Circuits:
- inclusion
- non-inclusion
- combined

Changes:
- make existing public inputs private
- hash public inputs into one hash chain
- make hash chain the only public input

* chore: refactor batched trees and merkle tree metadata into separate crates

* feat: add proving keys for mt 32 (#1406)

* feat: add support for circuits with height 32 and update related tests

* chore: remove combined_32_0_0 verifying key file

* refactor: remove deprecated verifying keys, restore inclusin_26_*.rs vkeys

* refactor: prover: integration tests for mt 32

* refactor: generate_keys.sh

* fix: bounded vec mut deserialization from immutable reference

* feat: add migrate state instruction

* fix: tests

* fix: lint

* fix: system cpi tests

* fix: tests

* cleanup

---------

Co-authored-by: Sergey Timoshin <[email protected]>
Co-authored-by: ananas-block <[email protected]>

.github/actionlint.yaml

@@ -4,6 +4,7 @@ self-hosted-runner:
     - buildjet-2vcpu-ubuntu-2204
     - buildjet-4vcpu-ubuntu-2204
     - buildjet-8vcpu-ubuntu-2204
+    - buildjet-16vcpu-ubuntu-2204
 # Configuration variables in array of strings defined in your repository or
 # organization. `null` means disabling configuration variables check.
 # Empty array means no configuration variable is allowed.

.github/workflows/light-examples-tests.yml

@@ -37,10 +37,10 @@ jobs:
             sub-tests: '[
               "cargo test-sbf -p token-escrow -- --test-threads=1"
             ]'
-          - program: name-service-without-macros-test
-            sub-tests: '[
-              "cargo test-sbf -p name-service-without-macros -- --test-threads=1"
-            ]'
+          # - program: name-service-without-macros-test
+          #   sub-tests: '[
+          #     "cargo test-sbf -p name-service-without-macros -- --test-threads=1"
+          #   ]'
 
     steps:
       - name: Checkout sources

.github/workflows/light-system-programs-tests.yml

@@ -41,33 +41,19 @@ jobs:
       matrix:
         include:
           - program: sdk-test-program
-            sub-tests: '[
-              "cargo-test-sbf -p sdk-test -- --test-threads=1"
-            ]'
+            sub-tests: '["cargo-test-sbf -p sdk-test"]'
           - program: account-compression
-            sub-tests: '[
-              "cargo-test-sbf -p account-compression-test"
-            ]'
+            sub-tests: '["cargo-test-sbf -p account-compression-test"]'
           - program: light-system-program
-            sub-tests: '[
-              "cargo-test-sbf -p system-test -- --test-threads=1"
-            ]'
+            sub-tests: '["cargo-test-sbf -p system-test"]'
           - program: light-registry
-            sub-tests: '[
-              "cargo-test-sbf -p registry-test -- --test-threads=1"
-            ]'
+            sub-tests: '["cargo-test-sbf -p registry-test"]'
           - program: light-compressed-token
-            sub-tests: '[
-              "cargo-test-sbf -p compressed-token-test -- --test-threads=1"
-            ]'
+            sub-tests: '["cargo-test-sbf -p compressed-token-test"]'
           - program: system-cpi-test
-            sub-tests: '[
-              "cargo-test-sbf -p system-cpi-test -- --test-threads=1"
-            ]'
+            sub-tests: '["cargo-test-sbf -p system-cpi-test"]'
           - program: random-e2e-test
-            sub-tests: '[
-              "RUST_MIN_STACK=8388608 cargo-test-sbf -p e2e-test -- --nocapture --test-threads=1"
-            ]'
+            sub-tests: '["cargo-test-sbf -p e2e-test"]'
     steps:
       - name: Checkout sources
         uses: actions/checkout@v4
@@ -84,6 +70,7 @@ jobs:
         run: |
           source ./scripts/devenv.sh
           anchor build
+          npx nx build @lightprotocol/test-programs
 
       - name: ${{ matrix.program }}
         run: |