bug: testify subtests stopped working

[uroborosq]

Did you check docs and existing issues?

• I have read the documentation.
• I have searched the existing issues.

Neovim version (nvim -v)

0.10.3

Operating system/version

archlinux

Output from :checkhealth neotest-golang

neotest-golang: require("neotest-golang.health").check()

Requirements ~
- OK Binary 'go' found on PATH: /usr/bin/go
- OK Found go.mod file for /home/dmitriy.tikhomirov/desktop/control_path/pkg/cp_errors/errors_test.go in /home/dmitriy.tikhomirov/desktop/control_path/go.mod
- OK Treesitter parser for go is installed
- OK neotest is available
- OK nvim-treesitter is available
- OK nio is available
- OK plenary is available

DAP (optional) ~
- OK Binary 'dlv' found on PATH: /home/dmitriy.tikhomirov/.local/share/nvim/mason/bin/dlv
- OK dap is available
- OK dapui is available
- OK dap-go is available

Gotestsum (optional) ~
- OK Binary 'gotestsum' found on PATH: /home/dmitriy.tikhomirov/.local/share/go/bin/gotestsum
- OK Tests will be executed by gotestsum.

Sanitization (optional) ~
- OK Sanitization is disabled

Describe the bug

Testify subtests no more detected by neotest, only suites and methods.
Looks like it related to bd8e69e, because usually receiver for suites is s, but this fix forbid any other receivers except t

If i rename s to t, everything works as expected

Steps To Reproduce

1. Write a simple suite with such method

func (s *Suite) TestFoo() {
    s.Run("foo", func() {
        // do smth
    })
    s.Run("bar", func() {
        // do smth other
    })
}

2. Open test summary, TestFoo is detected, but subtests foo and bar are not
3. Also can change receiver to (t *Suite), when it works as expected

Expected Behavior

Subtests detected with any receiver name

Your Lua setup

[fredrikaverpil]

Thanks for reporting. Can you please check if suite.Run works for you?

func (suite *Suite) TestFoo() {
    suite.Run("foo", func() {
        // do smth
    })
    suite.Run("bar", func() {
        // do smth other
    })
}

[fredrikaverpil]

I haven't got the time right now, but if you wish you can try adding a test among the existing testify tests and then modify the queries in https://github.com/fredrikaverpil/neotest-golang/blob/main/lua/neotest-golang/features/testify/lookup.lua that mentions suite.

For example, you should be able to do something like this (two occurrences in the file):

- operand: (identifier) @suite_lib (#eq? @suite_lib "suite")
+ operand: (identifier) @suite_lib (#match? @suite_lib "^[suite|s]$")

[uroborosq]

Hello, thanks for answering.
Case with suite doesn't help unfortunately.
I changed all lines with operand @suite_lib to add s receiver as you provided, but without success either.

As I can understand these queries, testify/lookup.lua contains queries for parsing like "starter" function for testify, which let standart golang test mechanism start testify suite.

My problem is with detecting subtests (e.g. table tests) which are defined in query.lua at the root (almost).
There were some changes, which now allows using only t as a receiver in subtests

[fredrikaverpil]

Ah yes, I think I made a mistake earlier then. I need to dig into this and see what to do. I think I would like to limit the use of s.Run and potential other variants to when you have the option testify_enabled = true only.

[fredrikaverpil]

@uroborosq would you mind giving the testify-operand branch a try?

{
  "fredrikaverpil/neotest-golang",
  branch = "testify-operand",
}

I think this might work. I just need to add some tests into #280 to verify it.

[uroborosq]

@fredrikaverpil ok, thanks I will try in the nearest future

[uroborosq]

Yes, it works but only for usual tests like

s.Run("foo", func() {
 // smth
})

but unfortunately doesn't work for table tests with such structure

cases := []testCase{ {name: "foo" ... }}
for _, test := range cases {
    s.Run(test.name, func() {
    ...
    }
}
}

[fredrikaverpil]

Table tests are not something I ever added support for, when it comes to testify. They are only supported for "vanilla" tests. Are you saying table tests used to work with testify? 🤔

The current testify specific tree-sitter queries would have to be extended to support table tests. I'm open to PRs which would enable this, but I don't plan on implementing this myself, as I don't use testify.

[uroborosq]

Table tests had been working good enough until bd8e69e

There is no any significant differences between vanilla testing and testify table tests in my opinion, just using suite receiver instead of testing t, so yes, it worked

[fredrikaverpil]

Okay, interesting. I can probably re-introduce this behavior fairly simply in that case.

It's somewhat funny that all of that was a side effect of having the base treesitter queries mixed with the testify treesitter queries. It's probably a good idea to not mix them and instead have two different set of queries so to avoid future bugs.

I'll look into it.

[fredrikaverpil]

@uroborosq I added back the table tests support but only for this kind of table tests:

func (s *ExampleTestSuite) TestSubTestOperand3() {
	tt := []struct {
		name  string
		value int
	}{
		{
			name:  "foo",
			value: 5,
		},
		{
			name:  "bar",
			value: 5,
		},
		{
			name:  "baz",
			value: 5,
		},
	}
	for _, tc := range tt {
		s.Run(tc.name, func() {
			s.VariableThatShouldStartAtFive = tc.value
			assert.Equal(s.T(), 5, s.VariableThatShouldStartAtFive)
		})
	}
}

Maybe you can see if this works for you, by using the testify-operand branch (#280)?

There are loads of other ones to copy from the "vanilla" query.lua file, but I will leave that for others to complete.

[fredrikaverpil]

@uroborosq take note of that I limited the operand to s (in s.Run). Maybe we can be more lenient here. What do you think?

EDIT: Hmm... I just realized you can't run the nearest test on the baz subtest. This is also what the behavior used to be like earlier. I'm not entirely sure why this is. Might have something to do with the testify-specific neotest positions tree modification. Unfortunately, supporting testify is not easy and I'm not sure I want the maintenance burden of having to support this, unless someone else is willing to help out. I'm not sure I want to merge these last changes (table tests) in. There might be other cases where this causes unforeseen issues.

[fredrikaverpil]

@uroborosq I'm releasing the initial fix you verified for me. But I'm holding off on the table tests one for now.

[uroborosq]

It seems that now it works as it was before from the first look:

• non-table subtests are working
• table tests with anonimous struct as you provided are working
• table test witn named struct are working either for some reason (it's a most common type of tests in my work project)

Run nearest test is also working for me

Speaking about different operands - s is pretty ok for me, we don't have another operands, because its suite, so s is logical)

May be you can do this configurable and leave s as default to keep it as much flexible as possible

[fredrikaverpil]

It seems that now it works as it was before from the first look:

• non-table subtests are working
• table tests with anonimous struct as you provided are working
• table test witn named struct are working either for some reason (it's a most common type of > tests in my work project)

That's nice to hear. Were you referring to the latest v1.9.2 release from today, or the testify-operand branch?

May be you can do this configurable and leave s as default to keep it as much flexible as possible

Hey, that's a great idea, I'll try to do that!