Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use BenchmarkCI to automatically detect major runtime regressions #923

Open
mfherbst opened this issue Dec 9, 2023 · 3 comments · May be fixed by #929
Open

Use BenchmarkCI to automatically detect major runtime regressions #923

mfherbst opened this issue Dec 9, 2023 · 3 comments · May be fixed by #929
Labels
testing Test system and test improvements

Comments

@mfherbst
Copy link
Member

mfherbst commented Dec 9, 2023

Since this runs in the CI we need to be a little conservative on what we time, some ideas:

Systems:

  • Something with many k-points but low Ecut
  • Something with many symmetries
  • Something with few k-points and higher Ecut

Tasks:

  • A Hamiltonian application
  • A single SCF step
  • Density computation and symmetrisation
  • Basis construction
  • A Sternheimer solve
  • A very crude response solve / derivative computation using AD

Probably with timing an SCF step and a single application of ε^\dagger for response on a not too trivial system we already do the key 80%.
@mfherbst mfherbst added the testing Test system and test improvements label Dec 9, 2023
@mfherbst
Copy link
Member Author

mfherbst commented Dec 9, 2023

Related to #542

@epolack
Copy link
Collaborator

epolack commented Dec 12, 2023 •
edited
Loading

I've just started working on that framework before seeing this issue. I can propose something if you have not yet started.
It would use PkgBenchmark, and I am also thinking about flexibility to run big humongous test-cases from your testproblems.
PkgBenchmark seems to be able to do the job easily coupled to @testitems. However if you want to look at older regressions (before a PR using PkgBenchmark and even before TestItemRunner), it is a bit more delicate.

@mfherbst
Copy link
Member Author

I have not yet worked on this, no. Feel free to propose something. I don't care so much about the backend. BenchmarkCI just looked like the least hassle and pretty neat to use in CI and locally.

I think most important is simplicity and flexibility, e.g. to just run basic tests like the ones I sketched above by dropping a julia runner script into a folder, but at the same time be able to only filter a subset for faster testing on github's CI.

I also bought a bigger machine recently, which we can use for larger periodic testing.

And of course regressions should be automatically annotated in PRs, so you don't miss them.

@epolack epolack linked a pull request Dec 12, 2023 that will close this issue
regression tests & benchmark #929
Draft
12 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
testing Test system and test improvements
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

regression tests & benchmark epolack/DFTK.jl
2 participants
@mfherbst @epolack