Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stability Test Improvement Ideas #66

Open
matthewfala opened this issue Apr 19, 2023 · 3 comments
Open

Stability Test Improvement Ideas #66

matthewfala opened this issue Apr 19, 2023 · 3 comments

Comments

@matthewfala
Copy link
Contributor

The following Stability Test improvements were discussed between Matt and Wesley on 4/18.

  • Execution-id: Include the execution name in the execution id
  • Execution-id date: ISO date should be readable in the execution name
  • Have an option to run individual test suites and cases
  • Report metrics have each day reporting from firelens datajet telling whether the test is alive
  • Run the stability tests metric crunch periodically and compute the Stability Test metrics on failure rate per testcase. Add alarms when failure rate goes above a certain number.
@PettitWesley
Copy link
Contributor

the technique here might be an interesting way for FLB to report that its alive, have it cat and EMF metric file with an exec input and send that: aws-samples/amazon-ecs-firelens-examples#111

@PettitWesley
Copy link
Contributor

Also we discussed:

  • test output should have two concepts: info needed to check results and audit log. Your audit log is all the config and the task ID and the cluster name and everything- that should go to S3. I think the test launcher should also output the execution ID and name and may be region and S3 bucket links to a file which we can commit to the repo as a record. Then this output is also used by the user to check the results later/create a doc on them.

@PettitWesley
Copy link
Contributor

Execute order 66 🚨

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants