Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: add performance script to PR jobs #3274

Open
wants to merge 53 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
1a6c0e6
feat: poc run performance against testenv
BlairCurrey Feb 5, 2025
f433b57
Revert "feat: poc run performance against testenv"
BlairCurrey Feb 5, 2025
0eeab9b
refactor: mv testenv to ./test from ./test/integration
BlairCurrey Feb 5, 2025
1d06cee
refactor: mv MockASE to new test-lib
BlairCurrey Feb 5, 2025
2ef91de
chore: generate gql
BlairCurrey Feb 5, 2025
fbad4e2
feat(performance): make environment configurable
BlairCurrey Feb 6, 2025
713cd81
feat(performance): make run and test script configurable per env
BlairCurrey Feb 7, 2025
07095f9
fix(performance): up liquidity, matching localenv
BlairCurrey Feb 7, 2025
b7022d2
ci: add performance test
BlairCurrey Feb 7, 2025
da89299
chore: fix eslint error
BlairCurrey Feb 7, 2025
6c33b43
chore: debug test in ci
BlairCurrey Feb 7, 2025
001eda5
chore: debug test in ci
BlairCurrey Feb 7, 2025
6f52bc9
chore: debug test in ci
BlairCurrey Feb 7, 2025
4388003
chore: debug test in ci
BlairCurrey Feb 7, 2025
38ab11b
chore: debug test in ci
BlairCurrey Feb 7, 2025
2789afd
chore: debug test in ci
BlairCurrey Feb 7, 2025
f29a751
chore: debug test in ci
BlairCurrey Feb 7, 2025
e8b09b8
fix: script typo
BlairCurrey Feb 7, 2025
143ab10
chore: debug test in ci
BlairCurrey Feb 7, 2025
81a1a12
chore: debug test in ci
BlairCurrey Feb 7, 2025
ea8e812
chore: debug test in ci
BlairCurrey Feb 7, 2025
c7587e7
chore: debug test in ci
BlairCurrey Feb 7, 2025
036b023
chore: debug test in ci
BlairCurrey Feb 7, 2025
f27d966
chore(auth): gql generate
BlairCurrey Feb 7, 2025
a6efa1a
chore: test manually commenting logs from perf test
BlairCurrey Feb 11, 2025
896ffd9
chore: test comment w/ summary, not logs
BlairCurrey Feb 11, 2025
0b834fe
Merge branch 'main' into bc/3240/performance-ci
BlairCurrey Feb 11, 2025
55bb709
fix: pr comment
BlairCurrey Feb 11, 2025
432c443
chore: try linking perf results to gh job
BlairCurrey Feb 11, 2025
87f74fe
Revert "chore: try linking perf results to gh job"
BlairCurrey Feb 11, 2025
19d5135
chore: add logs to pr comment, try link again
BlairCurrey Feb 11, 2025
da88616
chore: fix indentation
BlairCurrey Feb 11, 2025
aa99873
chore: try forming link
BlairCurrey Feb 11, 2025
ca57b9d
chore: rm badish link
BlairCurrey Feb 12, 2025
fc23590
fix: owner undefined
BlairCurrey Feb 12, 2025
3ce7591
chore: test ci
BlairCurrey Feb 12, 2025
d051dd6
chore: test ci
BlairCurrey Feb 12, 2025
1d0b50b
chore: test ci
BlairCurrey Feb 12, 2025
b8e2316
chore: clarify job script
BlairCurrey Feb 12, 2025
8a358db
fix: override test config in ci
BlairCurrey Feb 12, 2025
a7ce6c9
ci: test failure
BlairCurrey Feb 12, 2025
3d7f998
chore: ignore test summary
BlairCurrey Feb 12, 2025
e498465
ci: un-fail step
BlairCurrey Feb 12, 2025
747aa83
ci: tune test config
BlairCurrey Feb 12, 2025
c5b279d
ci: tune test config
BlairCurrey Feb 12, 2025
4a923db
ci: tune test config
BlairCurrey Feb 12, 2025
27d9d81
fix: 401 bug for duplicate reqs
BlairCurrey Feb 13, 2025
a829619
ci: tune test config
BlairCurrey Feb 13, 2025
f17acd8
docs: update performance readme
BlairCurrey Feb 13, 2025
cd83210
docs: update README
BlairCurrey Feb 13, 2025
128160c
docs: rm inaccurate info
BlairCurrey Feb 13, 2025
248ee43
ci: tune test config
BlairCurrey Feb 13, 2025
b910e1a
fix: return to orginal duration
BlairCurrey Feb 13, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
129 changes: 129 additions & 0 deletions .github/workflows/node-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,135 @@ jobs:
- name: Run tests
run: pnpm --filter integration run-tests

performance-test:
runs-on: ubuntu-22.04
needs: prerequisite
timeout-minutes: 5
steps:
- name: Checkout code
uses: actions/checkout@v4

- name: Setup environment
uses: ./.github/workflows/rafiki/env-setup

- name: Set up k6
uses: grafana/setup-k6-action@v1

- name: Setup hosts
run: |
echo "127.0.0.1 cloud-nine-wallet-test-backend" | sudo tee -a /etc/hosts
echo "127.0.0.1 cloud-nine-wallet-test-auth" | sudo tee -a /etc/hosts
echo "127.0.0.1 happy-life-bank-test-backend" | sudo tee -a /etc/hosts
echo "127.0.0.1 happy-life-bank-test-auth" | sudo tee -a /etc/hosts

- name: Build dependencies
run: pnpm --filter performance build:deps

- name: Bundle
run: pnpm --filter performance bundle

- name: Start test environment
run: pnpm --filter performance testenv:compose up --wait --build -d

- name: Start Mock Account Entity Services
run: nohup pnpm --filter performance start-mases > mases.log 2>&1 &

- name: Run performance tests
id: perf_test
run: |
pnpm --filter performance run-tests:testenv -k --vus 4 --duration 1m | tee performance.log
Copy link
Contributor Author

@BlairCurrey BlairCurrey Feb 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do we want the test config to be?

I opted for 4VUs and 1M after testing some different values. I think we want to:

  1. not make the duration longer than necessary. Just dont want to extend the time it takes CI to finish if we can help it. So I prefered more VUs for a shorter time to some extent. There may still be some room to increase duration if we want. This is taking ~3m while the docker builds are taking 4-5m
  2. reliably complete the tests without failures as a baseline. But approaching the point where we do see issues. I saw some errors at 5VUs so I backed off.

I think the hardware is determined by runs-on: ubuntu-22.04 and should always be the same across runs.

continue-on-error: true # Don't fail here, allows showing logs

- name: Print MASE logs
if: always()
run: cat mases.log

- name: Post/Update Performance Test Results as PR Comment
Comment on lines +195 to +205
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a grafana/k6 action that runs the tests and exports to grafana cloud and leaves a comment here with a link:
https://grafana.com/blog/2024/07/15/performance-testing-with-grafana-k6-and-github-actions/

I opted for doing it more manually as a start - was simpler to get going. And even with that action we might need to manual write/edit a comment if we want the actual metrics in the PR comment (looks like the comment just leaves a link). But that could probably be mitigated by setting good thresholds (fail if iterations < x etc. ) and just use the job for pass/fail and venture into grafana when it fails for more details.

It does appear like the free tier is fairly generous and it probably gives us a lot of stuff for free like a centralized view of the history of runs. Open to reworking if we want that but I think this gives us what we're looking for for now.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the comment generated: #3274 (comment)

It just upserts with the latest results and test logs.

if: always()
uses: actions/github-script@v7
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
Comment on lines +205 to +210
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am in-lining the javascript here because I found it easier than passing everything from the actions context in and it was just easier for development. We can move this to a file and just run it with node if we prefer though.

const fs = require('fs');
const summaryPath = './test/performance/k6-test-summary.txt';
const logPath = './performance.log';
let summaryContent = '';
let logContent = '';

// Read the k6 summary file
if (fs.existsSync(summaryPath)) {
summaryContent = fs.readFileSync(summaryPath, 'utf8');
} else {
summaryContent = 'Performance test summary not found.';
}

// Read the full logs
if (fs.existsSync(logPath)) {
logContent = fs.readFileSync(logPath, 'utf8');
} else {
logContent = 'Performance log not found.';
}

// Limit in case large file size to avoid error posting comment (65kb limit)
// https://github.com/orgs/community/discussions/41331
const maxLogSize = 60000;
const truncatedLogs = logContent.length > maxLogSize ? `...(truncated)...\n${logContent.slice(-maxLogSize)}` : logContent;


const commentBody = `
### 🚀 Performance Test Results

${summaryContent}

<details>
<summary>📜 Logs</summary>

\`\`\`
${truncatedLogs}
\`\`\`

</details>
`;

const { owner, repo } = context.repo;
const prNumber = context.payload.pull_request?.number;

if (prNumber) {
const comments = await github.rest.issues.listComments({
owner,
repo,
issue_number: prNumber
});

// Identify existing comment via title
const existingComment = comments.data.find(comment => comment.body.includes('🚀 Performance Test Results'));

if (existingComment) { // upsert comment
await github.rest.issues.updateComment({
owner,
repo,
comment_id: existingComment.id,
body: commentBody
});
} else {
await github.rest.issues.createComment({
owner,
repo,
issue_number: prNumber,
body: commentBody
});
}
} else {
console.log("No PR detected, skipping comment.");
}

- name: Fail if performance tests failed
if: steps.perf_test.outcome == 'failure'
run: exit 1




node-build:
runs-on: ubuntu-latest
timeout-minutes: 5
Expand Down
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -63,3 +63,5 @@ packages/**/src/openapi/specs/schemas.yaml
packages/**/src/openapi/specs/auth-server.yaml
packages/**/src/openapi/specs/resource-server.yaml
packages/**/src/openapi/specs/wallet-address-server.yaml

k6-test-summary.txt
Loading
Loading