Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve E2E Test Failure Handling and Reporting #1191

Merged
merged 21 commits into from
Oct 11, 2024
Merged

Improve E2E Test Failure Handling and Reporting #1191

merged 21 commits into from
Oct 11, 2024

Conversation

shammowla
Copy link
Collaborator

Description

  1. Test Retries: Added 3 retries for failed tests in Sauce Labs configuration to reduce flaky test failures.
  2. Failure Logging: Implemented a step to collect and store test results, especially for failed tests.
  3. GitHub Issue Creation: Automated the process of creating GitHub issues for failed tests.
  4. Issue Updates: For existing issues, the system now adds details about new failures to the issue.

Changes:

  • Updated .sauce/prod-safari.yml, .sauce/prod-firefox.yml, and .sauce/prod-chrome.yml to include 3 retries for failed tests.
  • Added a "Get Saucelabs Test Results" step in the workflow to collect test results.
  • Created a new script handle-test-failures.js to manage the creation and updating of GitHub issues for test failures.
  • Updated the "Handle Test Failures" step in the workflow to use the new script for each browser job.

Related Issue

Motivation and Context

Screenshots (if appropriate):

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Improvement (non-breaking change which does not add functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • I have signed the Adobe Open Source CLA or I'm an Adobe employee.
  • I have made any necessary test changes and all tests pass.
  • I have run the Sandbox successfully.

@shammowla shammowla merged commit 49395c8 into main Oct 11, 2024
175 checks passed
@shammowla shammowla deleted the wf-debug branch October 11, 2024 16:40
@jonsnyder jonsnyder added the ignore-for-release Do not include this PR in release notes label Oct 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ignore-for-release Do not include this PR in release notes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants