Grouping and extra output makes hard to find the error line

[ssbarnea]

While trying to evaluate use of this plugin for one project where I have a chain of 4 tox environments run one after another, I discovered two problems with it, which makes it a blocker for adoption.

• Undesired extra output is present that is not included in collapsed groups, mainly the 4 lines around recreate/installdeps/develop-inst/installed. These take more like 12-15 lines of console output due to the longer list of dependencies
• The errored environment is already collapsed and user needs to find its line and expand it. That is bit of the finding the needle in the hay-stack (console), which requires scrolling few screens.

I suppose that some of these are part of the grouping limitation on GHA, but even so we should make a feature request to GHA to allow groups to determine their default state when the group is ended (that is the moment when you know if there was a failure or not on that group).

Still, including the extra lines into the group is clearly under tox/plugin responsibility and we should ensure that messages related to specific group are not logged outside the group.

Please note the extra lines on screenshot below, see the scrollbar size, user is expected to scroll ~2 screens up to find the broken group (which is not even highlighted), expand it and scroll does several more screens in order to find the real test failure. That is a clear regression from the case where we manually had on GHA step for each tox environment, as the failed steps are already marked as red and they contain all the output.

I hope this report will be useful for improving this action in the future.

[ymyzk]

Thanks for the feedback!

Undesired extra output is present that is not included in collapsed groups, mainly the 4 lines around recreate/installdeps/develop-inst/installed. These take more like 12-15 lines of console output due to the longer list of dependencies

This is actually a known issue (#39). tox doesn't offer a good plugin hook to enable log line grouping where we actually want. We may be able to solve this by sending a feature request to tox.

The errored environment is already collapsed and user needs to find its line and expand it. That is bit of the finding the needle in the hay-stack (console), which requires scrolling few screens.

I think we cannot make the specific group opened by default using the current GitHub Actions API: https://github.com/actions/toolkit/blob/main/docs/commands.md#group-and-ungroup-log-lines As you suggested, this may be something we can request to GH.

Due to the reasons above, I cannot ensure that I can fix these problem soon. One mitigation I can offer is adding an option to disable log line grouping for users who don't want to use this at this point. How do you think?

[ssbarnea]

While I am confident that tox would be able to address this issue quickly, or at least quickly in tox v4 branch, I cannot say the same thing about github actions.

I think that what we need to do as soon as possible is to file a feature requires to github that explain clearly the "business case" for declaring a section as collapsed or open when you close the section block (not when you open it). That may be little bit confusing of HTML developers, as in HTML you never have attributes when you close a tag, but that is a requirement for something that is streamed like ANSI. Github moves considerably slower so the faster we inform them the better.

I do not think we need any changes yet, but not all features are implemented with local code changes, sometimes is only about asking others to enable you to do it.

PS. Please do not close the bug until we have links to the two external dependencies. If you close it after or not is up to you. Having it opened here is very important for making others find it and likely vote on those upstream feature requests.