Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve Continuous Integration via Witness-generation only tests #87

Closed
pgebheim opened this issue Mar 5, 2024 · 5 comments
Closed
Assignees
Labels
ci enhancement New feature or request

Comments

@pgebheim
Copy link
Contributor

pgebheim commented Mar 5, 2024

  • official Ethereum tests in the zk_evm CI (witness generation only): either hardcoded or random
  • some concrete blocks (either L1 or John's chain) to test the decoder parsing logic + witness generation
  • setting up an infra for regular testing and regression catch (we'd need to define the frequency, the types of run, the tests to be run)
@pgebheim pgebheim added the enhancement New feature or request label Mar 5, 2024
@pgebheim pgebheim changed the title Update CI to include smoke tests Improve Continuous Integration via Witness-generation only tests Mar 5, 2024
@Nashtare
Copy link
Collaborator

Nashtare commented Mar 6, 2024

I think some of those may be better suited for the zk_evm repo CI, as this is where PRs touching the logic are critical.

@pgebheim
Copy link
Contributor Author

pgebheim commented Mar 6, 2024

That's probably true -- I'll transfer this to the other repo.

@pgebheim pgebheim transferred this issue from 0xPolygonZero/zero-bin Mar 6, 2024
@github-project-automation github-project-automation bot moved this to Backlog in Zero EVM Mar 6, 2024
@pgebheim pgebheim moved this from Backlog to In Progress in Zero EVM Mar 6, 2024
@pgebheim pgebheim added this to the Type 1 milestone Mar 6, 2024
@Nashtare
Copy link
Collaborator

Nashtare commented Jun 7, 2024

This can probably wait until we merge the feat/cancun branch, or be targeted directly against this branch, given that this is going to happen soon and would break all hardcoded tests.

One thing that may be nice if running all tests takes too much time could be to have a CI job picking at random N tests from the Ethereum test suite (they can be preprocessed as done here), and persisting them upon failure until they're being cleared out for reproduceability, somewhat like how proptests work.

On current develop, blacklisting tests that are too expensive to run (either insane gas consumed or insame # ops), the remaining ~13k tests were taking ~2h to run on my M2 pro (for witness generation). Given that no parallelism was involved, we could reasonably have the CI check a few hundreds per run, if not more by using a self-hosted runner + adding multi-threading.

@Nashtare Nashtare added the ci label Sep 11, 2024
@temaniarpit27 temaniarpit27 moved this from In Progress to Backlog in Zero EVM Oct 6, 2024
@temaniarpit27 temaniarpit27 moved this from Backlog to Todo in Zero EVM Oct 8, 2024
@temaniarpit27 temaniarpit27 moved this from Todo to Backlog in Zero EVM Oct 16, 2024
@temaniarpit27
Copy link
Contributor

@Nashtare
I think this task can be covered with the following tasks: #726 #704 as I discussed with ben last time
Let me know your thoughts

@temaniarpit27 temaniarpit27 moved this from Backlog to In Progress in Zero EVM Oct 31, 2024
@temaniarpit27 temaniarpit27 moved this from In Progress to Backlog in Zero EVM Oct 31, 2024
@Nashtare
Copy link
Collaborator

Agreed, closing this one.

@Nashtare Nashtare closed this as not planned Won't fix, can't repro, duplicate, stale Oct 31, 2024
@github-project-automation github-project-automation bot moved this from Backlog to Done in Zero EVM Oct 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci enhancement New feature or request
Projects
Status: Done
Development

No branches or pull requests

4 participants