Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bench: add back benchmark and profiling infrastructure #3889

Open
wants to merge 19 commits into
base: main
Choose a base branch
from

Conversation

EdmundGoodman
Copy link
Collaborator

Summary

This PR moves the existing benchmarks and profiling infrastructure from https://github.com/xdslproject/xdsl-bench into the main xDSL repo so they can be used more easily by developers. In addition to this, it restructures and extends them to provide more microbenchmarks, component-level benchmarks, and end-to-end benchmarks.

A full description of the motivate and structure is available in benchmarks/README.md.

Future work

  1. Modify https://github.com/xdslproject/xdsl-bench to run against new benchmark repo (blocked by merging this PR)
  2. Add further benchmarks, with main areas for development being (to be done as needed within the provided framework)
    • Evaluate additional components, such as the printer and verifier
    • Evaluate larger "hero" end-to-end workloads, such as CIRCT, ASL, or quantum
    • Make synthetic data driving microbenchmarks more representative
  3. Consider reducing the amount of MLIR duplicated from the test directory, by leveraging the mlir-opt installation in CI to generate it on the fly

@EdmundGoodman EdmundGoodman added the bench Benchmark related changes label Feb 12, 2025
@EdmundGoodman EdmundGoodman self-assigned this Feb 12, 2025
@superlopuh
Copy link
Member

Is there a way to avoid duplicating the files? I was expecting this to be more minimal

Copy link

codecov bot commented Feb 12, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 91.26%. Comparing base (dab099d) to head (d577040).
Report is 3 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #3889   +/-   ##
=======================================
  Coverage   91.26%   91.26%           
=======================================
  Files         466      466           
  Lines       57949    57949           
  Branches     5575     5575           
=======================================
  Hits        52886    52886           
  Misses       3635     3635           
  Partials     1428     1428           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@EdmundGoodman
Copy link
Collaborator Author

Is there a way to avoid duplicating the files? I was expecting this to be more minimal

If I remove the benchmarks which parse/lex over all test MLIR files instead of individual ones that should cut down the size a lot?

@EdmundGoodman
Copy link
Collaborator Author

OK, this looks a bit more sensible in the number of files changed and shouldn't impact the expressiveness of the tests too much.

@compor compor changed the title Edmundgoodman/benchmark dev bench: add back benchmark and profiling infrastructure Feb 13, 2025

.PHONY: asv
asv:
cd .. && uv run asv run
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this .. feels a bit fragile, especially repeated at every make target.

Can't we use the --directory option along with exposing this as a make var along the whole Makefile?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bench Benchmark related changes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants