Feature idea: Condition for retrying tests #1761
Replies: 4 comments
-
Seems reasonable. I'd prefer to gate this only on exit codes to avoid heuristics that can go wrong. Would that be good enough for you? |
Beta Was this translation helpful? Give feedback.
-
Since in this case, the test gets a |
Beta Was this translation helpful? Give feedback.
-
I have the execution side of this mostly working -- I started writing a parser for this but realized that it'll likely conflict with some other work going on in #1159. Hopefully we'll figure out a way to sequence this work soon. |
Beta Was this translation helpful? Give feedback.
-
Oh, the mentioned PR has landed now! ^^ |
Beta Was this translation helpful? Give feedback.
-
We're considering switching to
nextest
to run our Rust tests in CI over at Ruffle. (Cc: ruffle-rs/ruffle#14461) It's been great so far!The final issue we have is, that compared to vanilla
cargo test
- perhaps because of the many processes starting and stopping - we are occasionally triggering what looks like a rare bug in the Ubuntu 22.04 graphics drivers.See for example: https://github.com/torokati44/ruffle/actions/runs/7254876905/job/19764476196#step:12:1448
We wouldn't want to allow retries/flakyness in general just because of this, because we don't actually have flaky tests, it's only the environment that's not ideal. And we would like to know if one of our tests actually becomes flaky, by having it fail.
Rather, what I think would be great for this, is if there was a way to say that we want to retry tests only if they failed because of this specific reason, but not otherwise.
For example, filtering conditions for this could be the exit code, or a substring/tail of the stdout/stderr output of the testing process.
Beta Was this translation helpful? Give feedback.
All reactions