gh-116322: Add Py_mod_gil module slot #116882
Conversation
|
I think it does make sense to land this now after all. The code won't function as intended until the GIL is disabled by default, but it will be nice to have the module slot available while I work on gh-116738, so I can tag modules as they're ready. This is structured such that there shouldn't be any behavioral differences until the GIL is disabled by default, except when a |
|
I'll take a look on Monday. |
|
A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated. Once you have made the requested changes, please leave a comment on this pull request containing the phrase |
|
@ericsnowcurrently I don't think there should be any code changes as a result of our big sub-thread. Please let me know if that's wrong! Before this is ready for another review, I need to make some other changes. It looks like the consensus in Discord is that we should enable the GIL when calling a module init function, because we don't know before the call if it's a single-phase init module that needs the GIL. Then if it's a multi-phase init module that provides The only nontrivial (but not too complicated) part of this will be adding a function to disable the GIL safely. It should also move the GIL-related logic up out of |
Yeah, I think you're right. |
More details: - Fix a race while enabling the GIL by checking if the GIL was enabled between a no-op call to `_PyEval_AcquireLock()` and the thread attaching, and trying again if it was. - Enable the GIL before running a module init function, since we can't know if it's a single-phase init module that doesn't support free-threading. Look at the state of the module after initialization to determine if it's safe to disable the GIL again. - Add `PyModule_SetGIL()`, which can be used by single-phase init modules to declare that they support running without the GIL. - Change `gil->enabled` from a simple on/off switch to a count of active requests to enable the GIL. This allows us to support multiple interleaved imports that each independently track whether the GIL should remain enabled. See the big comment in `pycore_ceval.h` for more details.
swtaarrs
requested review from
kumaraditya303,
brettcannon,
ncoghlan and
warsaw
as code owners
|
I have made the requested changes; please review again. This should be ready for review again. It grew in scope a bit, and now includes support for single-phase init modules (gh-117526) (some details are in the commit message). With the infrastructure needed to properly support multi-phase init modules, supporting single-phase init modules is just a few extra lines, but I can pull it out into a separate PR if anyone wants. One specific thing I'd like input on is the timing of the |
AlexWaygood
removed request for
serhiy-storchaka,
corona10,
pablogsal,
FFY00,
erlend-aasland,
kumaraditya303 and
AlexWaygood
|
!buildbot nogil |
|
🤖 New build scheduled with the buildbot fleet by @colesbury for commit 77d1652 🤖 The command will test the builders whose names match following regular expression: The builders matched are:
|
|
I found a few more modules that I missed during some testing, and will push a commit to add those shortly. |
|
!buildbot nogil |
|
🤖 New build scheduled with the buildbot fleet by @swtaarrs for commit d1fe0cc 🤖 The command will test the builders whose names match following regular expression: The builders matched are:
|
| @@ -609,6 +634,19 @@ state: | |||
|
|
|||
| .. versionadded:: 3.9 | |||
|
|
|||
| .. c:function:: int PyModule_ExperimentalSetGIL(PyObject *module, void *gil) | |||
There was a problem hiding this comment.
Ehm, can this please be renamed to PyUnstable as suggested before? We have such naming schemes for a reason.
There was a problem hiding this comment.
Sorry, I'll get a PR ready. I went with this name because having both Unstable and Experimental in the name felt redundant, and in this comment, @ericsnowcurrently expressed a preference for PyModule_ExperimentalSetGIL() over PyUnstable_Module_SetGIL().
Does anyone else have strong feelings on whether it should be PyUnstable_Module_ExperimentalSetGIL() or PyUnstable_Module_SetGIL()?
This PR adds the ability to enable the GIL if it was disabled at interpreter startup, and modifies the multi-phase module initialization path to enable the GIL when loading a module, unless that module's spec includes a slot indicating it can run safely without the GIL. PEP 703 called the constant for the slot `Py_mod_gil_not_used`; I went with `Py_MOD_GIL_NOT_USED` for consistency with pythongh-104148. A warning will be issued up to once per interpreter for the first GIL-using module that is loaded. If `-v` is given, a shorter message will be printed to stderr every time a GIL-using module is loaded (including the first one that issues a warning).
| @@ -249,6 +249,9 @@ _PyModule_CreateInitialized(PyModuleDef* module, int module_api_version) | |||
| } | |||
| } | |||
| m->md_def = module; | |||
| #ifdef Py_GIL_DISABLE | |||
There was a problem hiding this comment.
Should be Py_GIL_DISABLED.
This means: "Do not re-enable the GIL when importing rl.readline." See python/cpython#116322 See python/cpython#116882
This PR adds the ability to enable the GIL if it was disabled at interpreter startup, and modifies the multi-phase module initialization path to enable the GIL when loading a module, unless that module's spec includes a slot indicating it can run safely without the GIL.
PEP 703 called the constant for the slot
Py_mod_gil_not_used; I went withPy_MOD_GIL_NOT_USEDfor consistency with gh-104148.A warning will be issued up to once per interpreter for the first GIL-using module that is loaded. If
-vis given, a shorter message will be printed to stderr every time a GIL-using module is loaded (including the first one that issues a warning).📚 Documentation preview 📚: https://cpython-previews--116882.org.readthedocs.build/