forked from flutter/flutter
-
Notifications
You must be signed in to change notification settings - Fork 1
Adding a new Test Shard
Ian Hickson edited this page Jun 30, 2021
·
6 revisions
Further documentation on Flutter's build infrastructure can be found in https://github.com/flutter/flutter/blob/master/dev/bots/README.md.
A general outline of the requirements that a Flutter CI test shard has:
- On LUCI, test shards map to builders. Each test shard must have its own LUCI builder. For the Framework, these are defined in framework_config.star. Generally you will need to have both a pre-submit ("try" in LUCI terminology) builder and a post-submit ("prod") builder.
- This LUCI builder will specify a "recipe" to run. These are starlark scripts that determine the actual CI steps to run, and are defined in flutter.googlesource.com/recipes. Most Framework tests use the flutter/flutter_drone.py recipe. To learn how to edit these, see https://github.com/flutter/flutter/blob/master/dev/bots/README.md#editing-a-recipe.
- Builders are then added to .ci.yaml. These files are read by Flutter's build dashboard, and are used for scheduling builds.
It is important to land these changes in order to prevent any failing builds during the migration period:
- Framework tests are run by a Dart test runner called test.dart that lives in the framework repository. Any new test shards must first be added to this file. Merge this framework change.
- Next, add pre-submit ("try") and post-submit ("prod") builders to framework_config.star. More detailed instructions in the infra README. Ensure that the "shard" and "subshard"/"subshards" properties match what was added to test.dart in the previous step. Merge this infra change.
- Update .ci.yaml in the Framework tree to include the newly added builder. Verify that the entry is marked as
bringup: true
. New shards should always be marked in bringup to verify they are passing on master before being able to block the tree. When opening this PR, a pre-submit check for the new test shard should be triggered. Merge this framework change. - Monitor the CI results of the new shard on the Flutter build dashboard. After 10 consecutive passing builds, you can remove the
bringup: true
parameter from.ci.yaml
in the Framework tree. This will allow the test to block the tree, preventing breakages.
- Home of the Wiki
- Roadmap
- API Reference (stable)
- API Reference (master)
- Glossary
- Contributor Guide
- Chat on Discord
- Code of Conduct
- Issue triage reports
- Our Values
- Tree hygiene
- Issue hygiene and Triage
- Style guide for Flutter repo
- Project teams
- Contributor access
- What should I work on?
- Running and writing tests
- Release process
- Rolling Dart
- Manual Engine Roll with Breaking Commits
- Updating Material Design Fonts & Icons
- Postmortems
- Setting up the Framework development environment
- The Framework architecture
- The flutter tool
- API Docs code block generation
- Running examples
- Using the Dart analyzer
- The flutter run variants
- Test coverage for package:flutter
- Writing a golden-file test for package:flutter
- Setting up the Engine development environment
- Compiling the engine
- Debugging the engine
- Using Sanitizers with the Flutter Engine
- Testing the engine
- The Engine architecture
- Flutter's modes
- Engine disk footprint
- Comparing AOT Snapshot Sizes
- Custom Flutter engine embedders
- Custom Flutter Engine Embedding in AOT Mode
- Flutter engine operation in AOT Mode
- Engine-specific Service Protocol extensions
- Crashes
- Supporting legacy platforms
- Metal on iOS FAQ
- Engine Clang Tidy Linter
- Why we have a separate engine repo
- Reduce Flutter engine size with MLGO
- Setting up the Plugins development environment
- Setting up the Packages development environment
- Plugins and Packages repository structure
- Plugin Tests
- Contributing to Plugins and Packages
- Releasing a Plugin or Package
- Unexpected Plugins and Packages failures