Output on windows doesn't match correctly

[shonfeder]

We are seeing a bit of strange behavior in our CI tests only on windows runners.

Given this minimal example:

<!-- !test program
bash -c ">&2 echo foo"
-->

<!-- !test in blah -->
    junk

<!-- !test err blah -->
    foo

The test succeeds on MacOS and Linux (and on my local machine, running Ubuntu) but it fails on windows with

not ok 1 blah: error mismatch
  ---
  expected stderr: |
    foo

  actual stderr: |
    foo

  program: |
    bash -c ">&2 echo foo"
  stdout: ''
  input location: |
    line 21
  error location: |
    line 24
  program location: |
    lines 16-18

Comparison with stdout works as expected.

[shonfeder]

My mistake: stdout also fails to match correctly.

<!-- !test program
bash -c "echo foo"
-->

<!-- !test in blah -->
    junk

<!-- !test out blah -->
    foo

fails with

  expected stdout: |
    foo

  actual stdout: |
    foo

  program: |
    bash -c "echo foo"
  stderr: ''
  input location: |
    line 21
  output location: |
    line 24
  program location: |
    lines 16-18

Again, only on windows.

[shonfeder]

Just to make sure it wasn't some problem with bash, I've I tried the example use node from your readme. This also fails on windows:

<!-- !test in simple example -->

    console.log('a')
    console.log(42)
    console.log([1, 2, 3])

The output is:

<!-- !test out simple example -->

    a
    42
    [ 1, 2, 3 ]

fails with

TAP version 13
1..24
not ok 1 simple example: output mismatch
  ---
  expected stdout: |
    a
    42
    [ 1, 2, 3 ]

  actual stdout: |
    a
42
[ 1, 2, 3 ]

  program: |
    node
  stderr: ''
  input location: |
    lines 20-22
  output location: |
    lines 28-30
  program location: |
    line 16

[anko]

Hi, and thanks for the report.

I suspect the issue may concern line breaks. Windows uses CR LF (\r\n), while Linux uses just LF (\n). Since the characters are invisible, this would explain outputs that look like they should match but don't.

Txm internally checks test programs' stdout and stderr against the given expected values in the file with a simple !== operator. It doesn't attempt to standardise line breaks.

So if you have a test file authored on Linux (or on Windows with an editor in Linux-line-break mode, possibly automatically detected due to what the file already contains), and the resulting markdown file's tests are then run on Windows, what might happen is each test program outputs Windows line-breaks, while the expected output in the file has Linux line-breaks. Txm compares them, sees that they differ, and the test fails.

I think the reason the readme example may be failing is that it was copied from the readme, which uses Linux line-breaks. To confirm, does the readme "simple example" test pass if you create a new file on Windows and manually type in the relevant parts of the file, or otherwise guarantee Windows line-breaks?

(I don't have a suitable Windows machine nearby to test on right now.)

[shonfeder]

Hi, and thanks for the pointers and the useful utility :)

I also suspected line the CRLF line break. I tried normalizing the text by stripping out carriage returns (e.g., using approaches like these) but it didn't help.

I have not tried writing the tests in windows, as I also don't have a suitable Windows machine.

However, at your suggestion, I have run the file through unix2dos and tried running that through our CLI. Unfortunately it didn't fix the problem.

I did discover one more strange thing:

This test fails (only on Windows):

<!-- !test program
bash -c "echo foo"
-->

<!-- !test in foo -->
    junk

<!-- !test out foo -->
    foo

But this test succeeds on all platforms:

<!-- !test program
echo foo
-->

<!-- !test in foo -->
    junk

<!-- !test out foo -->
    foo

Unfortunately, the problem doesn't seem to be isolated to just running things through bash. The cli tool we are trying to test fails these kinds of tests whether wrapped in a call to bash or not. 🤷

[anko]

This issue inspired me to write a new feature to make invisible control characters visible whenever they appear in diffs, using Unicode's Control Pictures block. That feels like it needed to be done anyway, but perhaps it also helps diagnosis here?

The change is live on npm, published in [email protected]. If you have an occasion to try it on these failing tests on Windows, I would be interested in the results.

[shonfeder]

I'll try this tomorrow! Thanks for the quick feature-add. This seems like it will be quite useful generally!

[shonfeder]

Great feature! You were spot on with the CR line endings:

not ok 1 non-existent file: error mismatch
  ---
  expected stderr: |
    error: file ../examples/non-existent.file does not exist␍

  actual stderr: |
    error: file ../examples/non-existent.file does not exist

  invisible characters in diff: |
    ␍ represents Carriage Return ("\r") [U+000d]
  program: |
    bash -
  stdout: ''
  input location: |
    line 24
  error location: |
    line 28
  program location: |
    lines 13-15
  exit location: |
    line 26
  ---

The reason my attempts to remediate weren't working before was that I assumed the \r was coming from the output in the windows environment, but it seems that it is instead coming from the expectation in the file, which, however, does not actually have the CR line endings! Somehow I guess node or the windows environment is inserting the CR when it's reading from the file on windows?

[agardnerIT]

I think I'm having the same issue. I have a windows machine available and am happy to be a guinea pig.

I'm using visual studio and have set the line endings to LF. Running the example, I get:

txm test3.md  
TAP version 13
1..1
not ok 1 simple example: output mismatch
  ---
  expected stdout: |
    a
42
[ 1, 2, 3 ]␍

  actual stdout: |
    a
42
[ 1, 2, 3 ]

  invisible characters in diff: '␍ represents Carriage Return ("\r") [U+000d]'
  program: 'node'
  stderr: ''
  input location: 'lines 10-12'
  output location: 'lines 18-20'
  program location: 'line 6'
  ---

# 0/1 passed
# FAILED 1

txm --version
8.2.0