unraisablehook enhancements

[graingert]

A number of unraisablehook plugin enhancements:

• set the unraisablehook as early as possible and unset it as late as possible, to collect the most possible unraisable exceptions (please let me know if there's an earlier or later hook I can use)
• call the garbage collector just before unsetting the unraisablehook, to collect any straggling exceptions
• collect multiple unraisable exceptions per test phase
• report the tracemalloc allocation traceback, if available!
• avoid using a generator based hook to allow handling StopIteration in test failures
• report the unraisable exception as the cause of the PytestUnraisableExceptionWarning exception if raised.
• compute the repr of the unraisable.object in the unraisablehook so you get the latest information if available, and should help with resurrection of the object

A lot of these enhancements could also apply to the threading.excepthook plugin and I'll try to do that in a followup

Could help with #10404 it doesn't fix the issue yet because the warnings filter is removed before the session finishes, so any warnings that occur during session/config teardown are not reported. This PR improves the situation because it's now possible to configure warnings as errors by setting the PYTHONWARNINGS environment variable and ensure all unraisable exceptions are collected

It may also be a good idea to add a flag that calls gc.collect() at the end of every test phase to help associate errors with tests - but probably you don't want to run this every time as it impacts performance

[graingert]

# test_demo.py
import asyncio
import pytest

async def my_task():
    pass

def test_scheduler_must_be_created_within_running_loop() -> None:
    with pytest.raises(RuntimeError) as _:
        asyncio.create_task(my_task())

This is an improvement it works now if you do: python -W error -m pytest test_demo.py because there's an issue where warnings filters are removed too early

[graingert]

These fixes probably need to be applied to the thread hooks

[graingert]

this should be covered on pypy

[Zac-HD]

Leave that comment in the code? More likely for future maintainers to see it that way 🙂

[graingert]

Ah what I mean to say is that the coverage check is failing for this branch, but it shouldn't and I don't know why

[Zac-HD]

Looks great overall! Some comments below, merge when you think it's ready.

[Zac-HD]

Leave that comment in the code? More likely for future maintainers to see it that way 🙂

[Zac-HD]

Do we want to change this now?

[graingert]

Yes, but I don't know how

[Zac-HD]

I haven't looked at the code at all, but maybe set a flag from the unraisable hook, check it where we choose the exit code, and exit-with-error if it's set?

[graingert]

I think I know how to do this, but I need to report the exception group traceback from pytest_session_finish then raise sys.exit(pytest.ExitCode.ERROR), then detect any absolutely straggling unraisables from the config callback and raise that as INTERNALERROR

I'm not sure how to do that though

[bluetech]

I don't think we have a way to issue warnings at this late point.

I'm think that perhaps configure/unconfigure is not the best hooks for this, as they're too early/late. The purpose of the plugin is to catch unraisables from user test code, not from pytest internals. So maybe sessionstart/sessionfinish are the more appropriate hooks for setting up & catching stragglers. At this point we still have session which I think we can use to issue the straggler warnings/messages.

[graingert]

We still want to catch unraisable exceptions caused by pytest internals and plugins. An object created in user code could stay referenced by pytest or a plugin and then only issue the unraisablehook call during the last teardown+ GC run

[bluetech]

Thanks, these are nice improvements.

I left some comments.

[bluetech]

Can we make this an UnraisableMeta as well, perhaps preserving the original unraisable just without the extra info?

[graingert]

this BaseException handling needs to be as simple as possible - the failures here are the sorts of thing relating to missing globals and things. I've refactored to make this branch only load locals

[graingert]

I just noticed that raising an exception in a callback to Config.add_cleanup doesn't allow subsequent cleanups to run. I'll fix this in another pr

[nicoddemus]

Overall great work, left some minor comments.

[nicoddemus]

Thanks @graingert!

[bluetech]

I would keep this empty line, I think it's a good delineation.

[bluetech]

I don't think we have a way to issue warnings at this late point.

I'm think that perhaps configure/unconfigure is not the best hooks for this, as they're too early/late. The purpose of the plugin is to catch unraisables from user test code, not from pytest internals. So maybe sessionstart/sessionfinish are the more appropriate hooks for setting up & catching stragglers. At this point we still have session which I think we can use to issue the straggler warnings/messages.

[nicoddemus]

Thanks a lot @graingert!

[graingert]

Here's the version for threading's hook: #13016