Skip to content

Commit

Permalink
Support benchmarking PRs
Browse files Browse the repository at this point in the history
  • Loading branch information
dholroyd committed Feb 29, 2024
1 parent e8288a7 commit 8e63d79
Show file tree
Hide file tree
Showing 3 changed files with 103 additions and 1 deletion.
72 changes: 72 additions & 0 deletions .github/workflows/pr-benchmark-upload-from-main.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
name: Track Benchmarks

on:
workflow_run:
workflows: [Run and Cache Benchmarks]
types:
- completed

jobs:
track_with_bencher:
if: github.event.workflow_run.conclusion == 'success'
runs-on: ubuntu-22.04
env:
BENCHER_PROJECT: h264-reader
BENCHER_ADAPTER: rust_iai_callgrind
BENCHER_TESTBED: ubuntu-22.04
BENCHMARK_RESULTS: benchmark_results.txt
PR_EVENT: event.json
steps:
- name: Download Benchmark Results
uses: actions/github-script@v6
with:
script: |
async function downloadArtifact(artifactName) {
let allArtifacts = await github.rest.actions.listWorkflowRunArtifacts({
owner: context.repo.owner,
repo: context.repo.repo,
run_id: context.payload.workflow_run.id,
});
let matchArtifact = allArtifacts.data.artifacts.filter((artifact) => {
return artifact.name == artifactName
})[0];
if (!matchArtifact) {
core.setFailed(`Failed to find artifact: ${artifactName}`);
}
let download = await github.rest.actions.downloadArtifact({
owner: context.repo.owner,
repo: context.repo.repo,
artifact_id: matchArtifact.id,
archive_format: 'zip',
});
let fs = require('fs');
fs.writeFileSync(`${process.env.GITHUB_WORKSPACE}/${artifactName}.zip`, Buffer.from(download.data));
}
await downloadArtifact(process.env.BENCHMARK_RESULTS);
await downloadArtifact(process.env.PR_EVENT);
- name: Unzip Benchmark Results
run: |
unzip $BENCHMARK_RESULTS.zip
unzip $PR_EVENT.zip
- name: Export PR Event Data
uses: actions/github-script@v6
with:
script: |
let fs = require('fs');
let prEvent = JSON.parse(fs.readFileSync(process.env.PR_EVENT, {encoding: 'utf8'}));
core.exportVariable("PR_HEAD", `${prEvent.number}/merge`);
core.exportVariable("PR_BASE", prEvent.pull_request.base.ref);
core.exportVariable("PR_DEFAULT", prEvent.pull_request.base.repo.default_branch);
core.exportVariable("PR_NUMBER", prEvent.number);
- uses: bencherdev/bencher@main
- name: Track Benchmarks with Bencher
run: |
bencher run \
--if-branch '${{ env.PR_HEAD }}' \
--else-if-branch '${{ env.PR_BASE }}' \
--else-if-branch '${{ env.PR_DEFAULT }}' \
--ci-number '${{ env.PR_NUMBER }}' \
--github-actions "${{ secrets.GITHUB_TOKEN }}" \
--token "${{ secrets.BENCHER_API_TOKEN }}" \
--err \
--file "$BENCHMARK_RESULTS"
28 changes: 28 additions & 0 deletions .github/workflows/pr-benchmark.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
name: Run and Cache Benchmarks

on:
pull_request:
types: [opened, reopened, synchronize]

jobs:
benchmark:
name: Run Benchmarks
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
- uses: cargo-bins/cargo-binstall@main

- run: sudo apt-get install valgrind
- run: cargo binstall --no-confirm [email protected]
- run: |
IAI_CALLGRIND_COLOR=never cargo bench --bench ci_bench > benchmark_results.txt
- name: Upload Benchmark Results
uses: actions/upload-artifact@v4
with:
name: benchmark_results.txt
path: ./benchmark_results.txt
- name: Upload GitHub Pull Request Event
uses: actions/upload-artifact@v4
with:
name: event.json
path: ${{ github.event_path }}
4 changes: 3 additions & 1 deletion benches/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,4 +16,6 @@ There are two benchmarks here,
- run using `cargo bench --bench ci_bench`

The latter benchmark is run from a github actions workflow on commits to the main branch and the benchmark results are
uploaded to [bencher.dev](https://bencher.dev/perf/h264-reader).
uploaded to [bencher.dev](https://bencher.dev/perf/h264-reader).

[![h264-reader - Bencher](https://api.bencher.dev/v0/projects/h264-reader/perf/img?branches=a53abf65-ea6e-482e-8c55-cf6726e77864&testbeds=19d2a260-47fc-44ec-b7f8-314d88408ce7&benchmarks=cc26ec97-55ef-43a5-860f-861ae847d8b3&measures=1dc590ca-477f-4477-8295-672b05d33086&start_time=1706605032000&end_time=1709197032000 "h264-reader")](https://bencher.dev/perf/h264-reader?key=true&reports_per_page=4&branches_per_page=8&testbeds_per_page=8&benchmarks_per_page=8&reports_page=1&branches_page=1&testbeds_page=1&benchmarks_page=1)

0 comments on commit 8e63d79

Please sign in to comment.