test: add benchmark GitHub workflow

[yonadaaa]

Adds a GitHub Action where we can run benchmarks on a remote machine and inspect the output.

[changeset-bot]

⚠️ No Changeset found

Latest commit: 239e805

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[alvrs]

Is there a way for us to configure the benchmark suite to report absolute numbers instead of relative ones? ("x seconds" vs "x times faster than y")

[alvrs]

Also curious if you looked into actions like this one https://github.com/benchmark-action/github-action-benchmark that have built-in support for reporting the benchmark results to a github-pages branch

[yonadaaa]

Is there a way for us to configure the benchmark suite to report absolute numbers instead of relative ones? ("x seconds" vs "x times faster than y")

Yes, it logs a full table of stats before "x times faster than y". I'll look into what the numbers mean and see if we can compress them down to something more legible:

[yonadaaa]

Also curious if you looked into actions like this one https://github.com/benchmark-action/github-action-benchmark that have built-in support for reporting the benchmark results to a github-pages branch

My impression is that most Actions only support certain benchmark tools (ie. not Vitest/tinybench). However github-action-benchmark does mention some custom biggerIsBetter that we can use as an adapter - let me try that too :)

[alvrs]

Yes, it logs a full table of stats before "x times faster than y". I'll look into what the numbers mean and see if we can compress them down to something more legible:

Ahh that's great. Somehow didn't find it in the action logs

[yonadaaa]

Ahh that's great. Somehow didn't find it in the action logs

I had --silent on some of the runs which I think only prints the "x > y" summary.

[holic]

💪

[holic]

I think if we do pnpm run --recursive bench, it'll only run packages in the workspace with the bench script

which means we won't have to remember to update this later if we add benchmarks to more packages