Reducing Test Flakiness

Flakiness issue has caused a large portion of the Flutter tree redness, and below workflow will be enforced to reduce flaky issues. The framework post-submit DeviceLab tests will be focused on in the beginning, and the logic will be extended to other host only tests in the future.

Preventing flaky tests

Adding a new DeviceLab test

DeviceLab tests are located under /dev/devicelab/bin/tasks. If you plan to add a new DeviceLab test, please follow

• Create a PR to add test files
  • Make sure an ownership entry is created for the test in TESTOWNERS file
• Enable the test in the staging env. first for validation - make sure 50 consecutive successful runs without any flakiness issue
  • How: add the new test to the appropriate platform in devicelab_staging_config.star
  • Monitor the test execution in the milo dashboard
• If no flakiness issue pops up, then enable the test in the prod env. and you will see the new test in the build dashboard.
  • How
    • Add the new test to the appropriate platform in devicelab_config.star
    • Enable the test to build dashboard by adding an entry in prod_builders.json

Detecting flaky tests

On a weekly basis, the tree gardener will scan through test execution statistics over the past 15 days and identify top flaky ones

• If there are any test builders whose Flaky Ratio >= 2%
  • Mark the tests as flaky by updating the entry in prod_builders.json.
  • Create a tracking bug if not existing in the bug pool.
    • The sub-team TL will be assigned by default for further triage/re-assign.
    • Sub-team labels will be added as well for better tracking.
    • P1 will be labeled
• If there is not any test builder whose Flaky Ratio >= 2%, then look for the top test builder whose Flaky Ratio < 2%
  • Mark the test as flaky by updating the entry in prod_builders.json.
  • Create a tracking bug if not existing in the bug pool.
    • The sub-team TL will be assigned by default for further triage/re-assign.
    • Sub-team labels will be added as well for better tracking.
    • P2 will be labeled

Fixing flaky tests

The TL will help triage, reassign, and attempt to fix the flakiness.

If fixed, the test can be re-enabled after being validated for 50 consecutive runs without flakiness issues (task without exclamation point in flutter build dashboard).

If not fixable, the test will be removed from the flutter build dashboard or deleted from CI completely depending on specific cases.