[4.1.1] ImportError: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process #694
Comments
|
I'm not familiar with how ceph is implemented, is there an obvious reason why it'd be initializing modules multiple times per process? Does it use subinterpreters? |
|
From my point of view, case should as similar as pyca/cryptography#9016 (comment). BTW now I could combine both Ceph 18.2.1 + python3-cryptography 41.0.7 + python3-bcrypt 4.0.1 without error; the error only happen when running with bcrypt >= 4.1.0. Therefore I guess bcrypt should reference some handling from cryptography between 41.0.1..41.0.7 for preventing multiple initialization? |
|
Hmm, so the problem requires using both cryptography 41.0.7 and bcrypt
4.1.1? This is very confusing, and I can't imagine what cross-package
interaction would be.
…On Tue, Dec 12, 2023 at 9:24 AM Wong Hoi Sing Edison < ***@***.***> wrote:
From my point of view, case should as similar pyca/cryptography#9016
(comment)
<pyca/cryptography#9016 (comment)>
.
BTW now I could combine both Ceph 18.2.1 + python3-cryptography 41.0.7 +
python3-bcrypt 4.0.1 without error; the error only happen when running with
bcrypt >= 4.1.0.
Therefore I guess bcrypt should reference some handling from cryptography
between 41.0.1..41.0.7 for preventing multiple initialization?
—
Reply to this email directly, view it on GitHub
<#694 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAAGBBWCEFUH7YKSX35JLLYJBSKDAVCNFSM6AAAAABAQ2VAX2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJSGEZTOMBYGA>
.
You are receiving this because you commented.Message ID:
***@***.***>
--
All that is necessary for evil to succeed is for good people to do nothing.
|
|
The case with Ceph 18.2.1 are:
Original report for Ceph 18.2.1 + cryptography 41.0.x could be found from:
BTW as above cross check shown, now cryptography no longer become the root cause, but coming from bcrypt ;-( |
|
Is it straightforward for me to reproduce this locally, or do I need a full ceph installation? |
|
I have a Vagrant Box for Ceph 18.2: https://app.vagrantup.com/alvistack/boxes/ceph-18.2 It is now rebuilding with my latest functional combination (i.e. cryptography 41.0.7 + bcrypt 4.0.1): https://gitlab.com/alvistack/vagrant-ceph/-/pipelines/1103864469 If you are using VirtualBox (see https://github.com/alvistack/vagrant-ceph#quick-start): Once the box is up and running:
|
|
If I built a custom wheel, would you be able to test that out and see if you can reproduce? |
Or you may provide your forked GitHub branch, so I could keep rebuilding and testing with it in my local dev env? |
|
#695 -- you can either download a wheel from the wheel-builder job, or you can build yourself from that branch. |
|
@alex my quick check and report:
For the last case, the new error message is now: See PyO3/pyo3#576 |
|
Hmm, on the one hand, if ceph really is using subinterpreters, there's nothing we can do. On the other hand, the fact that this error is showing up only with certain version combinations suggestions there's a deeper bug here. I'm afraid I don't have a suggestion other than that we really need a minimal reproducer here, something smaller than all of ceph. |
In order to avoid `ImportError: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process` introduced since pyo3 >= 0.17.0, which only allow each #[pymodule] to be initialized once, this PR temporarily rollback to pyo3 = 0.16.6. It also remove the use of `PyUserWarning` which introduced since pyo3 > 0.18.0. See PyO3/pyo3@78ba70d See PyO3/pyo3@1d20f2a Fixes pyca#694 Signed-off-by: Wong Hoi Sing Edison <[email protected]>
In order to avoid `ImportError: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process` introduced since pyo3 >= 0.17.0, which only allow each #[pymodule] to be initialized once, this PR temporarily rollback to pyo3 = 0.16.6. It also remove the use of `PyUserWarning` which introduced since pyo3 > 0.18.0. See PyO3/pyo3@78ba70d See PyO3/pyo3@1d20f2a Fixes pyca#694 Signed-off-by: Wong Hoi Sing Edison <[email protected]>
In order to avoid `ImportError: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process` introduced since pyo3 >= 0.17.0, which only allow each `#[pymodule]` to be initialized once, this PR temporarily rollback to pyo3 = 0.16.6. It also remove the use of `PyUserWarning` which introduced since pyo3 > 0.18.0. See PyO3/pyo3@78ba70d See PyO3/pyo3@1d20f2a Fixes pyca#694 Signed-off-by: Wong Hoi Sing Edison <[email protected]>
In order to avoid `ImportError: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process` introduced since pyo3 >= 0.17.0, which only allow each `#[pymodule]` to be initialized once, this PR temporarily rollback to pyo3 = 0.16.6. It also remove the use of `PyUserWarning` which introduced since pyo3 > 0.18.0. See PyO3/pyo3@78ba70d See PyO3/pyo3@1d20f2a Fixes pyca#694 Signed-off-by: Wong Hoi Sing Edison <[email protected]>
In order to avoid `ImportError: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process` introduced since pyo3 >= 0.17.0, which only allow each `#[pymodule]` to be initialized once, this PR temporarily rollback to pyo3 = 0.16.6. It also remove the use of `PyUserWarning` which introduced since pyo3 > 0.18.0. See PyO3/pyo3@78ba70d See PyO3/pyo3@1d20f2a Fixes pyca#694 Signed-off-by: Wong Hoi Sing Edison <[email protected]>
|
Ceph really does use subinterpreters. It also used to work on my system (F39) after a patch to remove My suspicion is the check for multiple instances itself is/was somehow flawed and doesn't always go off (or used not to), but Ceph has always been using subinterpretrs. Ceph bug: https://tracker.ceph.com/issues/63529 |
|
Yes. I have, in effect, vendored a rebuild of python-bcrypt with: diff --git a/src/_bcrypt/Cargo.toml b/src/_bcrypt/Cargo.toml
index a9c7f7c..02317c8 100644
--- a/src/_bcrypt/Cargo.toml
+++ b/src/_bcrypt/Cargo.toml
@@ -6,7 +6,7 @@ edition = "2018"
publish = false
[dependencies]
-pyo3 = { version = "0.20.0", features = ["abi3"] }
+pyo3 = { git = "https://git.st8l.com/luxolus/pyo3", tag = "v0.20.3-subint+1", features = ["abi3", "unsafe-allow-subinterpreters"] }
bcrypt = "0.15"
bcrypt-pbkdf = "0.10.0"
base64 = "0.21.5"https://git.st8l.com/luxolus/pyo3/commit/338c71d0ad10f7ae38b7b44e576d49b91ed20d99 Which is possible because this python module doesn't store |
|
For reference, in case anyone's stumbling over this issue, adding sub-interpreter support in PyO3 is being worked on over here: PyO3/pyo3#3451 The Ceph folks that commented here (hi again!) are already aware of this; just wanted to leave some breadcrumbs for others to follow in case anyone's interested in helping out as well. |
|
One possible work-around is to simply disable the error check that is causing this, and to allow unsafe operation. This is done in this commit: zer0def/pyo3@eba5243 I've made a collection of a small subset of the various distros and users affected by this bug; search engines show a bottomless network of discussion. pyca/cryptography#12080 The final comment is this:
I conclude there are three or 4 solutions:
All of these are unpleasant and distasteful solutions. Based on the quoted comment, I'm thinking that Ceph's use of python subinterpreters was a fundamental design flaw, and that meanwhile PyO3 is just broken and has been unwilling/unable to fix whatever design issue that they have with subinterpreters. It's been 14-15 months since this surfaced on the zillions of issue trackers and support forums, and there's been no forward progress. It's a halt-dead-stop blocker issue for ceph-dashboard, which apparently no one uses, based on general discussions on assorted forums. |
|
Hi guys. |
Similar as pyca/cryptography#9016, my ceph-mgr in 18.2.1 get error message once upgrade from bcrypt 4.0.1 to 4.1.1:
This was initially introduced by:
#[pymodule]to be initialized once, when pyo3 >= 0.17.0My only workaround for now is, revert back to bcrypt == 4.0.1, which still depends on pyo3 = { version = "0.15.2" }
The text was updated successfully, but these errors were encountered: