-
Notifications
You must be signed in to change notification settings - Fork 1
Writing a golden file test for package:flutter
(This page is referenced by comments in the Flutter codebase.)
If you want to learn how to write a golden test for your package, see the matchesGoldenFile
API docs. This wiki page describes the special process specifically for the Flutter team itself.
Golden file tests for package:flutter
use Flutter Gold for baseline and version management of golden files. This allows for golden file testing on Linux, Windows, MacOS and Web, which accounts for the occassional subtle rendering differences on between these platforms. If you have questions about Flutter Gold, cc @Piinks on your pull request.
- Creating a New Golden File Test
- Updating a Golden File Test
- Flutter Gold Login
flutter-gold
Checkreduced-test-set
tag
Write your test as a normal test, using testWidgets
and await tester.pumpWidget
and so on.
Put a RepaintBoundary
widget around the part of the subtree that you want to verify. If you don't, the output will be a 2400x1800 image, since the tests by default use an 800x600 viewport with a device pixel ratio of 3.0. If you would like to further control the image size, put a SizedBox
around your RepaintBoundary
to set constraints.
Add an expectation along the following lines:
await expectLater(
find.byType(RepaintBoundary),
matchesGoldenFile('test_name.subtest.subfile.png'),
);
The argument to matchesGoldenFile
is the filename for the screen shot. The part up to the first dot should exactly match the test filename (e.g. if your test is widgets/foo_bar_test.dart
, use foo_bar
). The subtest
part should be unique to this testWidgets
entry, and the part after that should be unique within the testWidgets
entry. This allows each file to have multiple testWidgets
tests each with their own namespace for the images, and then allows for disambiguation within each test in case there are multiple screen shots per test.
Golden tests may be executed locally on Linux, MacOS, and Windows platforms. All reference images can be found at Flutter Gold baselines. Some tests may have multiple golden masters for a given test, to account for rendering differences across platforms. The parameter set for each image is listed in each image digest to differentiate renderings.
Once you have written your test, run flutter test --update-goldens test/foo/bar_test.dart
in the flutter
package directory (where the filename is the relative path to your new test). This will update the images in bin/cache/pkg/skia_goldens/packages/flutter/test/
; the directories below that will match the hierarchy of the directories in the test
directory of the flutter
package. Verify that the images are what you expect; update your test and repeat this step until you are happy with the image.
When running flutter test
locally without the --update-goldens
flag, your test will pass, as it does not yet have a baseline for comparison on the Flutter Gold dashboard. The test will be recognized as new, and provide output for validation.
When you are happy with your image, you are ready to submit your PR for review. The reviewer should also verify your golden image(s), so make sure to include the golden(s) in your PR description.
New test results will be compiled into a tryjob on the Flutter Gold dashboard, under ChangeLists. There you will see your pull request and the associated golden files that were generated. Gold will also leave a comment on your pull request linking to your image results.
New tests can be triaged from these tryjobs, which will cause the pending flutter-gold
check to pass. Review the tryjob and the images that were generated, making sure they look as expected. Currently, we generate images for Linux, Mac, Windows, and Web platforms. It is common for there to be slight differences between them. Click the checkmark to approve the change, completing triage.
And that’s it! Your new golden file(s) will be checked in as the baseline(s) for your new test(s), and your PR will be ready to merge. 🎉
If renderings change, then the golden baseline in Flutter Gold will need to be updated.
When developing your change, local testing will produce a failure along with visual diff output. This visual diff will be generated using the golden baseline from Flutter Gold that matches the current paramset of your testing environment (currently based on platform). This allows for quick iterations and validation of your change. Locally, this test will not pass until the new baseline has been checked in.
When you are happy with your golden change, you are ready to submit your PR for review. The reviewer should also verify your golden image(s), so make sure to include the golden changes you have made in your PR description. Changes to tests will be compiled into a tryjob on the Flutter Gold dashboard, under ChangeLists. There you will see your pull request and the associated golden files that were generated, as well as the visual differences between the pre-existing baselines. Gold will also leave a comment on your pull request linking to your image results.
The updated tests can be triaged from these tryjobs, which will cause the pending flutter-gold
check to pass. Review the tryjob and the images that were generated, making sure they look as expected. Currently, we generate images for Linux, Mac, Windows and Web platforms. It is common for there to be slight differences between them. Click the checkmark to approve the change, completing triage.
And that’s it! Your new golden file(s) will be checked in as the baseline(s) for your new test(s), and your PR will be ready to merge. 🎉
Triage permission is currently restricted to members of flutter-hackers. For more information, see Contributor Access. Once you have been added as an authorized user for Flutter Gold, you can log in through the homepage of the Flutter Gold dashboard and proceed to triage your image results under Changelists.
The flutter-gold
check is applied to pull requests in flutter/flutter that execute golden file tests and are ready for review.
As golden file tests are run across multiple test shards, this check waits for all other tests to complete before checking to see if Gold received new images. While awaiting test completion, and in the event there are image changes, the flutter-gold
check will hold a pending state. This is primarily due to how the auto-roller used by flutter/engine works (context: https://github.com/flutter/flutter/issues/48744).
If there are no image changes, the flutter-gold
check will go green. If image changes were detected, a comment on your pull request will notify and provide a direct link to the images. Upon triaging, or approving, the images, the flutter-gold
check will go green within five minutes.
On some CI platforms in pre-submit, hermetic tests suites are not executed in order to conserve resources and expedite testing of other changes. To ensure that a golden file image is available for every platform, test files with golden tests are tagged with reduced-test-set
. This marks them for execution in these conservative test environments. Currently, framework tests on Mac and Windows platforms execute these reduced test sets. The analyzer will alert you if the tag is omitted from the test file. The tag should be formatted as such at the top of the file:
@Tags(<String>['reduced-test-set'])
For more context, see flutter.dev/go/reduce-ci-tests.
- Home of the Wiki
- Roadmap
- API Reference (stable)
- API Reference (master)
- Glossary
- Contributor Guide
- Chat on Discord
- Code of Conduct
- Issue triage reports
- Our Values
- Tree hygiene
- Issue hygiene and Triage
- Style guide for Flutter repo
- Project teams
- Contributor access
- What should I work on?
- Running and writing tests
- Release process
- Rolling Dart
- Manual Engine Roll with Breaking Commits
- Updating Material Design Fonts & Icons
- Postmortems
- Setting up the Framework development environment
- The Framework architecture
- The flutter tool
- API Docs code block generation
- Running examples
- Using the Dart analyzer
- The flutter run variants
- Test coverage for package:flutter
- Writing a golden-file test for package:flutter
- Setting up the Engine development environment
- Compiling the engine
- Debugging the engine
- Using Sanitizers with the Flutter Engine
- Testing the engine
- The Engine architecture
- Flutter's modes
- Engine disk footprint
- Comparing AOT Snapshot Sizes
- Custom Flutter engine embedders
- Custom Flutter Engine Embedding in AOT Mode
- Flutter engine operation in AOT Mode
- Engine-specific Service Protocol extensions
- Crashes
- Supporting legacy platforms
- Metal on iOS FAQ
- Engine Clang Tidy Linter
- Why we have a separate engine repo
- Reduce Flutter engine size with MLGO
- Setting up the Plugins development environment
- Setting up the Packages development environment
- Plugins and Packages repository structure
- Plugin Tests
- Contributing to Plugins and Packages
- Releasing a Plugin or Package
- Unexpected Plugins and Packages failures