Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tracking performance testing - collect and analyze results of benchmark runs #5464

Closed
thearusable opened this issue Sep 16, 2022 · 7 comments · May be fixed by kokkos/kokkos-benchmark-results#1
Closed

Tracking performance testing - collect and analyze results of benchmark runs #5464

thearusable opened this issue Sep 16, 2022 · 7 comments · May be fixed by kokkos/kokkos-benchmark-results#1
Assignees
@cz4rs @thearusable
Labels
Feature Request Create new capability; will potentially require voting

Comments

@thearusable
Copy link
Contributor

thearusable commented Sep 16, 2022 •
edited
Loading

Related to #5070 2.5(Propose a mechanism to track performance over time):

  • have a repo inside the kokkos organization where to store all the json files: let's call this the database-repo for now
  • push there (somehow) all the json files generated by the benchmark for example every time a PR is merged into main branch
  • have a script inside database-repo that automatically generates plots every time files are updated
  1. When the Performance Test will be run on CI we can add an additional step which will upload all generated json files to database-repo.
  1. After pushing benchmark results to database-repo there will be GitHub Action run in database-repo. The sole purpose of that action will be to run the analyzing software that will generate the required output data.
  2. To analyze the benchmark data we can use: https://github.com/bensanmorris/benchmark_monitor
  • By default this tool will generate chart with benchmark runs and HTML index file based on the Jinja2 template. This template could be modified to include benchmark context data.
  • Update: By default the images will be saved using following format BenchamrkName-metric.png so for example BM_SomeFunction-real_time.png.
  1. Generated files can be pushed using GitHub Actions to GitHub Pages. Github Action that can be used: https://github.com/marketplace/actions/deploy-to-github-pages

Example chart:
BM_SomeFunction-real_time

TODO:

  • I will create a quick demo on my repositories
@thearusable thearusable self-assigned this Sep 16, 2022
@thearusable thearusable added the Feature Request Create new capability; will potentially require voting label Sep 16, 2022
@thearusable thearusable changed the title Tracking performance testing - collect and analyze result of benchmark runs Tracking performance testing - collect and analyze results of benchmark runs Sep 16, 2022
@thearusable thearusable mentioned this issue Sep 16, 2022
Closed
6 tasks
@cz4rs
Copy link
Contributor

cz4rs commented Sep 16, 2022

Looks like it should "just work", thumbs up for using preexisting components!

@fnrizzi
Copy link
Contributor

fnrizzi commented Sep 21, 2022

thanks, look like a good solution!
Some comments:

  • i think we should also find a way to save plots with some filename convention
  • we need to figure out if the propose solution is ok in terms of permissions etc , that is up to @crtrott @dalg24

after they give the ok, we can try to do the prototype

@thearusable
Copy link
Contributor Author

By default the images will be saved using following format BenchamrkName-metric.png so for example BM_SomeFunction-real_time.png.

If we want a different name format then we will need to modify the benchmark_monitor.py script.

@masterleinad
Copy link
Contributor

My biggest concern is that we are nowadays mostly interested in performance on GPU backends and that the machines we are running CI on don't produce good/consistent enough results for performance regression testing.

@thearusable
Copy link
Contributor Author

Github action on database-repo can be configured to create new charts after any commit on branch(main). So it can be triggered by a commit from CI or by commit made manually.

We can make a separate directories to store benchmark results from CI and from target machines/proper benchmark machines.

@cz4rs cz4rs self-assigned this Oct 26, 2022
@cz4rs
Copy link
Contributor

cz4rs commented Nov 30, 2022 •
edited
Loading

Current status:

TODO:

  • use commit hashes to identify builds in graphs (after #5348: Add git information to benchmark metadata #5463 gets merged edit: merged recently, work in progress)
  • use the metric that is prefixed with "FOM" (figure of merit) automatically
    • potentially: ensure that all the benchmarks contain such metric on kokkos side
  • make it easier to search for specific results

@cz4rs cz4rs added Enhancement Improve existing capability; will potentially require voting and removed Enhancement Improve existing capability; will potentially require voting labels Dec 7, 2022
This was referenced Jan 4, 2023
Merged
Open
@cz4rs
Copy link
Contributor

cz4rs commented May 24, 2023

Performance results are collected and stored in https://github.com/kokkos/kokkos-benchmark-results.

@cz4rs cz4rs closed this as completed May 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@cz4rs cz4rs

@thearusable thearusable

Labels
Feature Request Create new capability; will potentially require voting
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add benchmark results processing and deployment cz4rs/kokkos-benchmark-results
4 participants
@masterleinad @cz4rs @thearusable @fnrizzi