v1.11.0-RC1

[martin-schulze-vireso]

Due to the extensive changes necessary for the dynamic test registration, this is first released as a candidate for users to check if it works as intended.

Please provide your feedback on this release candidate here to improve the final release.

---

This discussion was created from the release v1.11.0-RC1.

[henning-schild]

Gave it a quick try and it seems to look good on gentoo. https://packages.gentoo.org/packages/dev-util/bats

But there i also just ran the internal test suite and nothing else, especially nothing dealing with dynamic registration

[abathur]

I did run into trouble with 3 tests breaking on file/dir not found errors in one of my test suites. I'm not certain what's causing the break, but it manifests coming as a shift in what my suite's expecting PWD to be. (I may just be holding it wrong in a way you don't want to support.)

The ~important failure case is around a test that I build a file for dynamically at test time in order to bake in an absolute path. Here's a diff of how I'm adapting the test(s) that fail in order to get them working again:

abathur/resholve@d870bdb#diff-1cd59227c5054dad716cc83a31abe52d4762512e24a4d145a5db33f3bb0c2aac

I'm less concerned about the other 2 cases at the end of the diff where I just removed $PWD/tests/ (I can't remember why I set these up this way and don't see an obvious reason to have done so. I probably just accidentally copy-pasted it around from the earlier test.

For context, my suite also uses the following setup/teardown functions: https://github.com/abathur/resholve/blob/3e7f0191d787184a5dbebdab7ad0b634d0d1b642/tests/helpers.bash#L1-L12

Edit: After sorting out a workaround that was suitable for RC1, I also tried running against current bats with the workaround in place--and this also produced a file not found error (but in a different location).

[martin-schulze-vireso]

I understand that you are most concerned about having to move echo "source $PWD/tests/source_present_target.sh" > $PWD/tests/temp_source_test.sh into the heredoc?

AFAIK, there is nothing in the new implementation that (persistently) changes directories, so you should always be on the same PWD as you started bats with. Your setup/teardown would only affect code in the test. (Previously, the call to setup was inside the test, this has been moved now, but this should not be related to your problem here.)

There is one difference that might be involved: The new code for gathering the tests being registered does an extra source of the file, so free code will be executed once more. However, my tests still show the same PWD for that execution...

[abathur]

I'm using the heredoc to hold multiple invocations that should all pass the same assertions, so it would be ideal to avoid pulling in stuff that isn't part of the invocation.

The error itself is:

not ok 8 allows exempted absolute source paths in 38ms
# (from function `_expect_status' in file tests/helpers.bash, line 17,
#  from function `require' in file tests/helpers.bash, line 156,
#  in test file tests/behavior.bats, line 89)
#   `})' failed
# status: 1
# output:
# /nix/store/kwx16dywlbas8pf72lz9b2vxv8ac9q69-bats-1.11.0-RC1/lib/bats-core/test_functions.bash: line 166: /private/tmp/nix-build-resholve-test.drv-0/tmp.NllpdUI3T0/tests/temp_source_test.sh: No such file or directory

The function at that line has been changed in the past few months, but it looks like the commit was included in 1.10.0 and I don't see the same problem there.

[martin-schulze-vireso]

After some digging I think I found the cause: In the old code, the setup was executed as first command of @test (injected by the preprocessor). The new version calls setup before calling the @test function. This means your heredoc is now executed after pushd in setup and receives the changed $PWD.

While this is a change of behavior, I a feel like this is a niche case and the new behavior is more consistent. (You would not want to be able to consume the heredoc stdin of the @test function in setup, right?)

Assuming were on the same page the question becomes, how to deal with this case? For your specific case I'd recommend to rewrite the specific test such that it does not rely on $PWD at all:

1. use $BATS_TEST_TPMDIR (only availbale in setup/@test)instead of rolling your own tmpdir, this relieves you of the cleanup burden as well
2. rely on $BATS_TEST_DIRNAME to get the source directory for copying the files independently of $PWD
3. move the temp_source_test.sh setup into the test; (I am thinking about offering per test setup overrides but that will still take some time)

like this (beware, untested code! setup would have to be adapted as well):

test "allows exempted absolute source paths" {
  echo "source $BATS_TEST_TMPDIR/source_present_target.sh" > "$BATS_TEST_TMPDIR/temp_source_test.sh"
  require <({
    status 0
    line -1 ends "/source_present_target.sh"
  }) <<CASES
resholve --interpreter $INTERP --keep source:$BATS_TEST_TMPDIR/source_present_target.sh < $BATS_TEST_TMPDIR/temp_source_test.sh && rm $BATS_TEST_TMPDIR/temp_source_test.sh
CASES
}