Multithreaded subinterpreters can be running during finalization

[ZeroIntensity]

Crash report

What happened?

(Found while investigating a fix for #125920.)

Subinterpreters that are running in a different thread can still be running upon the finalization of the main interpreter, depending on what the thread was doing. Take the following script:

from threading import Thread
import _interpreters


def silly():
    interp = _interpreters.create()
    _interpreters.run_string(interp, "import time; time.sleep(100)")


if __name__ == "__main__":
    Thread(target=silly).start()

On 3.13 and 3.14, killing the process with CTRL+C results in an assertion failure:

python: Python/pylifecycle.c:2480: finalize_subinterpreters: Assertion `!_PyInterpreterState_IsRunningMain(interp)' failed.

However, if the thread is daemon, then the interpreter fully segfaults:

from threading import Thread
import _interpreters


def silly():
    interp = _interpreters.create()
    _interpreters.run_string(interp, "import time; time.sleep(100)")


if __name__ == "__main__":
    Thread(target=silly, daemon=True).start()

I'm going to investigate possible fixes.

cc @ericsnowcurrently

CPython versions tested on:

3.13, CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

No response

Linked PRs

• gh-126016: Force crossinterpreter thread states to get cleaned up #126026

[rruuaanng]

Using .join() will not trigger an assertion error. For the reason, please see the PR submitted by Eric and me.

>>> def silly():
...     interp = _interpreters.create()
...     _interpreters.run_string(interp, "import time; time.sleep(100)")
...
>>> if __name__ == "__main__":
...     Thread(target=silly).start()
...
>>>
>>> Thread(target=silly).start()
>>> Thread(target=silly).start()
>>> quit


^CTraceback (most recent call last):
  File "/root/cpython/Lib/threading.py", line 1540, in _shutdown
    _thread_shutdown()
KeyboardInterrupt:
python: Python/pylifecycle.c:2463: finalize_subinterpreters: Assertion `!_PyInterpreterState_IsRunningMain(interp)' failed.

>>> def silly():
...     interp = _interpreters.create()
...     _interpreters.run_string(interp, "import time; time.sleep(100)")
...
>>> t = Thread(target=silly)
>>> t.start()
>>> t.join()
^CTraceback (most recent call last):
  File "<python-input-14>", line 1, in <module>
    t.join()
    ~~~~~~^^
  File "/root/cpython/Lib/threading.py", line 1092, in join
    self._handle.join(timeout)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^
KeyboardInterrupt

[ZeroIntensity]

Yes, I know. The issue is that CTRL+C'ing causes the thread to not finish joining, meaning that the thread state is still marked as running. I'm working on a fix to properly clean up thread states before the interpreter shuts down using pending calls, but daemon threads will probably have to get addressed elsewhere (there's a data race, I think).

[rruuaanng]

If you want, you can insert a signal catcher to catch SIGINT and do error handling on it. And in the error handling iterate over the list of threads and handle them

[ZeroIntensity]

That would result in a thread safety nightmare if the thread was still running :)

[ZeroIntensity]

#126026 addresses non-daemon threads. It's a bit hacky, though, because I have to manually mess with the status of the thread state.