ImportError: PyO3 modules may only be initialized once per interpreter process #12080
Comments
|
As discussed in that issue thread, supporting sub-interpreters requires PyO3/pyo3#3451 to be resolved in pyo3. |
|
OK, closing, and redirecting my attention there. |
|
Hmm. Is there a way to revert whatever it is that is breaking, until a fix becomes available for PyO3/pyo3#3451 ? The maintainer there is posting remarks about "issues in my personal life" and "now that college classes are in session" and "this is more complicated than I thought", coupled to it being 14 months since opened with apparently no actual progress, makes me think that a patch is years away, or might never arrive. In case that issue is never resolved, or only gets a fix years from now, is there perhaps some intermediate steps that can be taken in this project to work around that bug? Maybe just not use PyO3 at all, or not use sub-interpreters, something along those lines? |
|
No, there is no straightforward way to revert the relevant change.
…On Sun, Dec 1, 2024 at 2:23 PM Linas Vepštas ***@***.***> wrote:
Hmm. Is there a way to revert whatever it is that is breaking, until a fix
becomes available for PyO3/pyo3#3451
<PyO3/pyo3#3451> ? The maintainer there is
posting remarks about "issues in my personal life" and "now that college
classes are in session" and "this is more complicated than I thought",
coupled to it being 14 months since opened with apparently no actual
progress, makes me think that a patch is years away, or might never arrive.
In case that issue is never resolved, or only gets a fix years from now,
is there perhaps some intermediate steps that can be taken in this project
to work around that bug? Maybe just not use PyO3 at all, or not use
sub-interpreters, something along those lines?
—
Reply to this email directly, view it on GitHub
<#12080 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAAGBBQJGO5IP3CTBUPFCT2DNO2DAVCNFSM6AAAAABSZ7CRA6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMJQGIYTMNRVHA>
.
You are receiving this because you commented.Message ID:
***@***.***>
--
All that is necessary for evil to succeed is for good people to do nothing.
|
|
So the current plan is to stay broken, possibly forever? I don't get it. |
|
I wouldn't call it a plan, but yes, the reality is that unless someone steps up to do the work, sub-interpreters will remain unsupported. |
|
Who is using the sub-interpreters? Some blob of code somewhere in the stack decides "OK, start subinterpreter here". If I can locate that, then that would be the place to disable or bypass or mutex-lock or whatever. I'm coming at this from the Ceph side, as a Ceph user (not developer). In the past, this message seemed to be an anoying warning that merely clogged up the error logs, but could be safely ignored. Today, I'm actually trying to do something, and this is appears to be a blocker. I think. I'll double check to see if it really is blocking, or just an annoying warning. |
|
Ceph is making use of sub-interpreters. I do not know why. |
|
For the record, the stack trace I'm looking at is this: This looks like a straightforward naive API call to create a cert, with nothing fancy going on. The only "obvious" work-around would be some Search engines report that the Ceph variant of this is being discussed on a variety of forums. The generic work-around is "revert to an earlier version of the python crypto module". I'll see if I can get the Debian maintainers to roll back to an early 2023 version of crypto, but I imagine there will be pushback. Maybe the Ceph people can do something. Sigh. |
|
The relevant portion of the ceph code is at
https://github.com/ceph/ceph/blob/69c919838b4774559e407d3ce6a08d668c9b6f51/src/mgr/PyModule.cc#L282
…On Sun, Dec 1, 2024 at 2:58 PM Linas Vepštas ***@***.***> wrote:
For the record, the stack trace I'm looking at is this:
# ceph dashboard create-self-signed-cert
Error EINVAL: Traceback (most recent call last):
File "/usr/share/ceph/mgr/mgr_module.py", line 1811, in _handle_command
return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/share/ceph/mgr/mgr_module.py", line 474, in call
return self.func(mgr, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/share/ceph/mgr/dashboard/module.py", line 476, in set_mgr_created_self_signed_cert
cert, pkey = create_self_signed_cert('IT', 'ceph-dashboard')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/share/ceph/mgr/mgr_util.py", line 527, in create_self_signed_cert
from OpenSSL import crypto
File "/lib/python3/dist-packages/OpenSSL/__init__.py", line 8, in <module>
from OpenSSL import SSL, crypto
File "/lib/python3/dist-packages/OpenSSL/SSL.py", line 19, in <module>
from OpenSSL.crypto import (
File "/lib/python3/dist-packages/OpenSSL/crypto.py", line 21, in <module>
from cryptography import utils, x509
File "/lib/python3/dist-packages/cryptography/x509/__init__.py", line 6, in <module>
from cryptography.x509 import certificate_transparency
File "/lib/python3/dist-packages/cryptography/x509/certificate_transparency.py", line 10, in <module>
from cryptography.hazmat.bindings._rust import x509 as rust_x509
ImportError: PyO3 modules may only be initialized once per interpreter process
This looks like a straightforward naive API call to create a cert, with
nothing fancy going on. The only "obvious" work-around would be some if
(not already imported) import OpenSSL.crypto. Not clear if that will
work, but this kind of protection from multi-imports is common in python
code. I've seen it in ROS (robot operating system), and in Blender (3D
animation system) which have complex multi-module cascading everybody is
calling everything all the time in unpredictable order kinds of ways. Given
the complexity of Ceph, its probably equally uuuhh, "stochastic" (its got a
zillion daemons running on a zillion hosts, so like ROS)
Search engines report that the Ceph variant of this is being discussed on
a variety of forums. The generic work-around is "revert to an earlier
version of the python crypto module". I'll see if I can get the Debian
maintainers to roll back to an early 2023 version of crypto, but I imagine
there will be pushback. Maybe the Ceph people can do something. Sigh.
—
Reply to this email directly, view it on GitHub
<#12080 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAAGBDTJYKJKFPP4NPGSWT2DNTAFAVCNFSM6AAAAABSZ7CRA6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMJQGIZDQNJZHE>
.
You are receiving this because you commented.Message ID:
***@***.***>
--
All that is necessary for evil to succeed is for good people to do nothing.
|
|
Sigh. OK. Closing again. Here's a relevant redhat bug report: I'll do some noisy posting here as I find other relevant threads, so that the next person will, auhhh be saved some time tracking this down. |
Also-ran:
|
|
If the dep isn't needed then yes, that's a good solution. For future folks who may find this thread: subinterpreters were never explicitly supported by this project and when they "worked" they were untested and potentially unsound with our module. Subinterpreters have a long and checkered history in Python and projects that consume them ultimately need to either minimize their deps to only those who explicitly claim support or work with upstreams to create such support. The only way forward here is by assisting the pyo3 project in explicitly supporting subinterpreters. That may take the form of direct contribution, a variety of possible sponsorships, or simply reaching out to the pyo3 maintainers on their issue tracker to reopen discussion since it previously stalled. |
I agree this is the only true solution, but I will note that use of So, another solution is for pyo3 module maintainers to expressly opt out of the troublesome behavior (storing Python objects in static locations) in their own module design. I'm not necessarily suggesting this for this (or any other) module; but it is a solution that doesn't require waiting for pyo3 to land official subinterpreter support. It could even be an exercise for Ceph, to fork and modify as needed the module, then depend on (a vendored variant?) of that module. |
|
As soon as pyo3 has a "module state" API, we'll migrate our statics to that.
…On Mon, Dec 2, 2024 at 6:19 AM Paul Stemmet ***@***.***> wrote:
If the dep isn't needed then yes, that's a good solution. For future folks who may find this thread: subinterpreters were never explicitly supported by this project and when they "worked" they were untested and potentially unsound with our module. Subinterpreters have a long and checkered history in Python and projects that consume them ultimately need to either minimize their deps to only those who explicitly claim support or work with upstreams to create such support.
The only way forward here is by assisting the pyo3 project in explicitly supporting subinterpreters. That may take the form of direct contribution, a variety of possible sponsorships, or simply reaching out to the pyo3 maintainers on their issue tracker to reopen discussion since it previously stalled.
I agree this is the only true solution, but I will note that use of LazyPyImport statics (src/types.rs) will require a redesign in the presence of whatever changes pyo3 makes, given that storing python objects in a static is fundamentally undefined behavior in the presence of a 1:N process:interpreter architecture. Moreover, it is possible to use other pyo3 modules (like python-bcrypt) now as they provably do not store Py<'_> objects in statics in the rust code. I have used this to overcome the pyo3 pains for the Archlinux Ceph build, excluding that which relies on python-cryptography (and other static-using pyo3 infected modules).
So, another solution is for pyo3 module maintainers to expressly opt out of the troublesome behavior (storing Python objects in static locations) in their own module design.
I'm not necessarily suggesting this for this (or any other) module; but it is a solution that doesn't require waiting for pyo3 to land official subinterpreter support. It could even be an exercise for Ceph, to fork and modify as needed the module, then depend on (a vendored variant?) of that module.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.Message ID: ***@***.***>
--
All that is necessary for evil to succeed is for good people to do nothing.
|
|
Hi guys. |
There is "kind of" a way to make it work. It is easy to create self-signed certificates, there must be half-a-dozen or more tools that do this. What's missing is instructions for how to import such certs into Ceph. FWIW, you can even get CA-signed certs for free, from the "Let's Encrypt" project. |
|
This is in reference to issue #9016 which is marked resolved, closed and locked. Its currently December 2024, and this remains unfixed, i.e. an open issue, in both Debian Stable (bullseye) and Debian Testing. There's no additional information I can add to this issue report that is not already noted in issue #9016. There's no obvious work-around or patch available that can be deployed. So ???
The text was updated successfully, but these errors were encountered: