Add support for “known failures” to transcripts #5394
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Shows that unisonweb#5337 is _not_ fixed.
This is basically the opposite of `:failure` – it indicates that a successful result is incorrect. The correct result may either be an error (which will be caught by the transcript runner) or a different result then what is currently expected (which won’t be caught by the runner, but the appearance of `:incorrect` conveys to the programmer that the output.md diff may be an improvement from the previous state).
Shows that unisonweb#5178 is _not_ fixed.
sellout
force-pushed
the
transcript-known-failures
branch
from
9126ca4
to
56a2f5c
Compare
|
I wanna think a little more about what the label names should be, because I think |
|
I agree. It might be worth doing #5214 before this. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
This allows issues that haven’t yet been resolved to be reflected in the transcripts. This makes it easier to close tickets as soon as issues have been fixed, and avoids having to manually re-try transcripts from a ticket to see if the situation is the same.
Fixes #5350.
Implementation notes
This adds two new transcript tags
:failureand:incorrect. They behave similarly to:errorand no tag, respectively, except the error messages are different when the results don’t match.:failurerepresents a stanza that is intended to succeed, but currently fails (whereas:erroris intended to fail):incorrectrepresents a stanza that is intended to error, but currently doesn’t.Test coverage
This adds three new transcripts for open tickets exercising the two new tags.
Loose ends
There are lots of open tickets that should also get transcripts added.