Skip to content

Commit

Permalink
Make running Playwright to updatesnapshots easier (#4585)
Browse files Browse the repository at this point in the history
  • Loading branch information
madewithkode authored Jul 19, 2024
1 parent 8c1bc5b commit e6259b4
Show file tree
Hide file tree
Showing 3 changed files with 62 additions and 9 deletions.
35 changes: 29 additions & 6 deletions .github/workflows/ci_cd.yml
Original file line number Diff line number Diff line change
Expand Up @@ -799,11 +799,11 @@ jobs:
- storybook
include:
- name: playwright_vr
script: "test:playwright visual-regression"
script: "test:playwright visual-regression -u"
- name: playwright_e2e
script: "test:playwright e2e"
- name: storybook
script: "test:storybook"
script: "test:storybook -u"

steps:
- name: Checkout repository
Expand All @@ -819,13 +819,38 @@ jobs:
- name: Run Playwright tests
run: just frontend/run ${{ matrix.script }} --workers=2

- name: Check for changes after running tests with -u
id: check_snapshots_changed
if: failure() || success()
run: |
diff_names="$(git diff --binary --name-only)"
if [ -n "$diff_names" ]; then
git add . && git diff --staged --binary > /tmp/snapshot_diff.patch
echo "snapshots_changed=true" | tee "$GITHUB_OUTPUT"
printf "The following snapshots were updated:\n"
while IFS= read -r filename; do
printf "\t- %s\n" "$filename"
done <<< "$diff_names"
exit 1
else
echo "snapshots_changed=false" | tee "$GITHUB_OUTPUT"
fi
- uses: actions/upload-artifact@v4
if: failure()
id: test-results
with:
name: ${{ matrix.name }}_test_results
path: frontend/test-results

# This step only runs if there was a failure and snapshots indeed changed.
- uses: actions/upload-artifact@v4
if: failure() && steps.check_snapshots_changed.outputs.snapshots_changed == 'true'
id: snapshot-patch
with:
name: ${{ matrix.name }}_snapshot_diff
path: /tmp/snapshot_diff.patch

# This job runs when `playwright` doesn't and always passes, thus allowing
# PRs to meet the required checks criteria and be merged.
bypass-playwright:
Expand Down Expand Up @@ -885,11 +910,9 @@ jobs:
help_body<<$EOF
**Playwright failure test results**: <https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}>
It looks like some of the Playwright tests failed. You can download the trace output and image diffs for the failed tests under the "Artifacts" section in the above page.
You can read the [docs on how to use this artifact](https://docs.openverse.org/frontend/reference/testing_guidelines.html#debugging).
It looks like some of the Playwright tests failed. If you made changes to the frontend UI without updating snapshots, this might be the cause. You can download zipped patches containing the updated snapshots alongside a general trace of the tests under the "Artifacts" section in the above page. They're named in the form \`*_snapshot_diff\` and \`*_test_results\` respectively.
Additionally, if you made changes to the frontend UI, you may need to [update relevant playwright test snapshots](https://docs.openverse.org/frontend/guides/test.html#updating-snapshots).
You can read more [on how to use these artifacts in the docs](https://docs.openverse.org/frontend/reference/testing_guidelines.html#debugging).
If the test is flaky, follow the [flaky test triage procedure](https://docs.openverse.org/general/test.html#flaky-tests).
$EOF
Expand Down
1 change: 1 addition & 0 deletions docker/dev_env/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ RUN mkdir /pipx \
nodejs npm \
python3.12 pipx \
docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin \
unzip \
&& dnf clean all \
&& pipx install --global \
httpie \
Expand Down
35 changes: 32 additions & 3 deletions documentation/frontend/reference/testing_guidelines.md
Original file line number Diff line number Diff line change
Expand Up @@ -357,15 +357,44 @@ output of failed tests by finding the `trace.zip` for the relevant test in the
[Playwright Trace Viewer](https://playwright.dev/docs/trace-viewer) to inspect
these (or open the zip yourself and poke around the files on your own).
For Playwright tests failing in CI, a GitHub comment will appear with a link to
download an artifact of the `test-results` folder.
Playwright tests in CI are run with `-u` option by default, this means that
snapshots will automatically be updated for modified parts of the UI if
Playwright detects that. See
[Updating snapshots](/frontend/guides/test.md#updating-snapshots) for more
reading about this.
When this happens, a GitHub comment will appear with a link to download zipped
artifacts named in the form `*_snapshot_diff.zip`. Download and save this to the
repository root. Once downloaded, decompress and apply them to your working
branch by running:
`ov unzip -p *_snapshot_diff.zip | git apply`
The above command basically uses the `unzip` tool to unpack the contents of the
downloaded archive file named `*_snapshot_diff.zip`, the `-p` option prevents it
from actually creating any extracted file but rather prints the contents to the
standard output which is then piped to `git apply`.
```{note}
Remember to replace `*_snapshot_diff.zip` with the actual downloaded filename.
Keep in mind that you'd need to delete the downloaded file
(`*_snapshot_diff.zip`) from the repository after successfully applying
the changes to avoid committing them.
```
After successfully applying the patch, stage, commit and push the latest changes
to your branch upstream and you should most likely have Playwright CI tests
pass.
Visual regression tests that fail to match the existing snapshots will also have
`expected`, `actual` and `diff` files generated that are helpful for
understanding why intermittent failures are happening. These are generated
automatically by Playwright and will be placed in the `test-results` folder
under the fully qualified name of the test that failed (with every parent
describe block included).
describe block included). This is also available for download in the Artifacts
section after every failed Playwright test in the CI.
Additionally, you can run the tests in debug mode. This will run the tests with
a headed browser as opposed to a headless (invisible) one and allow you to watch
Expand Down

0 comments on commit e6259b4

Please sign in to comment.