[6.2.1 client, macOS, arm64] cannot use AES encryption: Error: cannot import python-cryptography

[psobolewskiPhD]

Describe the bug
Following the AES example usage (https://github.com/Xpra-org/xpra/blob/master/docs/Network/AES.md) trying to attach with 6.2.1 client on macOS arm64 (DMG downloaded from https://github.com/Xpra-org/xpra/tree/master#official-stable-downloads ) the connection fails with a traceback:

2024-10-31 16:52:05,987 Error: cannot import python-cryptography
2024-10-31 16:52:05,987 error preparing connection: invalid encryption 'AES', options: none
Traceback (most recent call last):
  File "/Users/sobolp/Applications/Xpra.app/Contents/Resources/lib/python/xpra/client/base/client.py", line 379, in send_hello
    hello = self.make_hello_base()
            ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sobolp/Applications/Xpra.app/Contents/Resources/lib/python/xpra/client/base/client.py", line 455, in make_hello_base
    cipher_caps = self.get_cipher_caps()
                  ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sobolp/Applications/Xpra.app/Contents/Resources/lib/python/xpra/client/base/client.py", line 472, in get_cipher_caps
    raise ValueError(f"invalid encryption {enc!r}, options: {csv(ciphers) or 'none'}")
ValueError: invalid encryption 'AES', options: none

If I don't use AES, everything works, but the statistics tab also results in spam Error: cannot import python-cryptography

To Reproduce
Steps to reproduce the behavior:

1. server command: xpra start --start=xterm \ --bind-tcp=0.0.0.0:10000,encryption=AES,keyfile=aes.txt (xpra v6.2.1-r0, ubuntu jammy, from official source)
2. client command: xpra attach "tcp://<ip>:10000/?encryption=AES&keyfile=./aes.txt"
3. specific action to trigger the bug: None, can't connect

System Information (please complete the following information):

• Server OS: [e.g. RHEL 8.x] Ubuntu Jammy
• Client OS: [e.g. Windows 10] MacOS 14.7
• Xpra Server Version [e.g. 4.2.3] : xpra v6.2.1-r0
• Xpra Client Version [e.g. 4.3] : v6.2.1-r0

Additional context
This used to work in March, so using xpra 5.0, although I needed to set: CRYPTOGRAPHY_OPENSSL_NO_LEGACY=1

[totaam]

Something must have changed in the latest version of python-cryptography or one of its dependencies.
Which means we also have to update the packaging scripts - and for all xpra versions, as this will affect newer v5.x builds just the same.

Here are the recent updates to python-cryptography only:

• 42.0.4
• 42.0.5
• 42.0.7
• 42.0.8
• 43.0.0
• 43.0.1
• 43.0.3

[totaam]

I can reproduce the problem simply by running Xpra.app/Contents/Helpers/Network_info -v.

crypto backend init failure
Traceback (most recent call last):
  File "/Users/macos12/Desktop/Xpra.app/Contents/Resources/lib/python/xpra/net/crypto.py", line 88, in crypto_backend_init
    backend = default_backend()
              ^^^^^^^^^^^^^^^^^
  File "cryptography/hazmat/backends/__init__.pyc", line 11, in default_backend
  File "cryptography/hazmat/backends/openssl/__init__.pyc", line 7, in <module>
  File "cryptography/hazmat/backends/openssl/backend.pyc", line 7, in <module>
ModuleNotFoundError: No module named 'cryptography.hazmat.bindings._rust'
Error: cannot import python-cryptography

And I get the same failure when running python ./xpra/net/net_util.py from a jhbuild shell, so it looks like something is fundamentally broken with the installation of python-cryptography's rust bindings.
There is no _rust module in my installation at all!

[totaam]

I tried with pip, but it moaned:

pyopenssl 24.0.0 requires cryptography<43,>=41.0.5, but you have cryptography 43.0.3 which is incompatible.

And I eventually landed here which led me to this comment: pyca/pyopenssl#1320 (comment)
Which means that pyopenssl and python-cryptography have to be updated in sync.
They were not, because pyopenssl was missing from the dependencies documentation - now fixed in d9d95f9

pyOpenSSL has now been updated in jhbuild to version 24.2.1: Xpra-org/gtk-osx-build@19ea82f - then another commit because of pypi URL breakage: Xpra-org/gtk-osx-build@a4534f8

But we're not done yet! Of course not.
It seems that the setuptools build script of python-cryptography doesn't actually build the rust bindings!
Oh joy.

cd src/rust
cargo build

This fails rather dramatically:

  = note: Undefined symbols for architecture x86_64:
            "_PyArg_UnpackTuple", referenced from:
                __cffi_f_ASN1_ENUMERATED_set in libcryptography_cffi-f49f72473c4b8b1e.rlib(c4d9326f651c9b94-_openssl.o)
                __cffi_f_ASN1_INTEGER_set in libcryptography_cffi-f49f72473c4b8b1e.rlib(c4d9326f651c9b94-_openssl.o)
                __cffi_f_ASN1_INTEGER_to_BN in libcryptography_cffi-f49f72473c4b8b1e.rlib(c4d9326f651c9b94-_openssl.o)
                __cffi_f_ASN1_STRING_to_UTF8 in libcryptography_cffi-f49f72473c4b8b1e.rlib(c4d9326f651c9b94-_openssl.o)
                __cffi_f_ASN1_TIME_set_string in libcryptography_cffi-f49f72473c4b8b1e.rlib(c4d9326f651c9b94-_openssl.o)
                __cffi_f_ASN1_TIME_to_generalizedtime in libcryptography_cffi-f49f72473c4b8b1e.rlib(c4d9326f651c9b94-_openssl.o)
                __cffi_f_BIO_get_mem_data in libcryptography_cffi-f49f72473c4b8b1e.rlib(c4d9326f651c9b94-_openssl.o)

Followed by another few hundred lines of similar missing Python symbols.
And rustup update did not help.

That's because of the new python packaging pep517. Lots of emojis in the output, but not very helpful with all the breakage it inflicts. Things were building just fine before this "improvement".

Now, this installs python-cryptography correctly on x86_64:

pip install --target  $PREFIX/lib/python3.11/site-packages .

(and how are we supposed to call this from jhbuild?)

But not on arm64:

maturin failed
        Caused by: MACOSX_DEPLOYMENT_TARGET is invalid

Which is fixed by setting this environment variable before calling pip:

export MACOSX_DEPLOYMENT_TARGET=12.0

And after all this, the net_util tool runs without complaints.
But the bundled application does not:

crypto backend init failure
Traceback (most recent call last):
  File "/Users/totaam/Desktop/Xpra.app/Contents/Resources/lib/python/xpra/net/crypto.py", line 88, in crypto_backend_init
    backend = default_backend()
              ^^^^^^^^^^^^^^^^^
  File "cryptography/hazmat/backends/__init__.pyc", line 11, in default_backend
  File "cryptography/hazmat/backends/openssl/__init__.pyc", line 7, in <module>
  File "cryptography/hazmat/backends/openssl/backend.pyc", line 7, in <module>
  File "cryptography/hazmat/bindings/_rust.pyc", line 13, in <module>
  File "cryptography/hazmat/bindings/_rust.pyc", line 9, in __load
  File "imp.pyc", line 343, in load_dynamic
ImportError: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process
Error: cannot import python-cryptography

Which looked like a known issue with PyO3: pyca/bcrypt#694 / pyca/cryptography#9016

But it's not!
I got a more helpful error message when trying the import from the bundled Python shell:

$ ./Desktop/Xpra.app/Contents/Helpers/Python          
Python 3.11.9 (main, Sep 14 2024, 17:51:18) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import cryptography
>>> from cryptography.hazmat.backends import default_backend
>>> default_backend()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "cryptography/hazmat/backends/__init__.pyc", line 11, in default_backend
  File "cryptography/hazmat/backends/openssl/__init__.pyc", line 7, in <module>
  File "cryptography/hazmat/backends/openssl/backend.pyc", line 7, in <module>
  File "cryptography/hazmat/bindings/_rust.pyc", line 13, in <module>
  File "cryptography/hazmat/bindings/_rust.pyc", line 9, in __load
  File "imp.pyc", line 343, in load_dynamic
ModuleNotFoundError: No module named '_cffi_backend'

And so the correct fix is a change to our packaging script instead.
And since I've already spent way too much time on this issue and I couldn't figure out how to make py2app bundle cffi and _cffi_backend properly, so I hacked it instead: be3f3ec

@psobolewskiPhD please try version 6.3-r36875 or later from the beta area

---

BTW, since you're using AES, be aware of some important issues - only some of which will be fully fixed in xpra 6.3: #4375

[psobolewskiPhD]

Wow, epic sleuthing.
Will test tomorrow.
To give more context, I'm using xpra in a singularity container in an internal network environement (SLURM cluster) with many users. So it's a pretty secure environment, but want to prevent people from "looking" into colleagues desktops. For now I've switched to using password with a uuidgen string.