Cannot connect with TLS secured NiFi 2.x using nipyapi.security.set_service_ssl_context and Python 3.12

[ChrisSamo632]

• Nipyapi version: 0.21.0
• NiFi version: 2.0.0-M4
• NiFi-Registry version: 2.0.0-M4
• Python version: 3.12
• Operating System: MacOS (running NiPyApi in a Linux based Docker Image)

Description

Connections to TLS-secured (with custom certificates) NiFi fails after setting nipyapi.security.set_service_ssl_context - trying to get the service status times out (with bool_response=True) or fairly quickly with a urllib3 error if bool_response=False:

Cannot create a client socket with a PROTOCOL_TLS_SERVER context

From a quick look online, it seems there have been changes to the urllib3 / ssl library setup through recent versions of Python, which is possibly causing the problem:

• https://stackoverflow.com/a/78863325
• Confusing behavior of ssl.create_default_context() python/cpython#96972

This worked for me when creating an ssl_context manually with Purpose.SERVER_AUTH instead of the Purpose.CLIENT_AUTH that NiPyApi uses when a cert/key are specified in the nipyapi.security.set_service_ssl_context call.

What I Did

• use SmallStep CA & CLI to create a custom CA and TLS certificates
• run NiFi using the generate TLS certificates as Keystore & Truststore
• create a user PKI set of certificates
• configure NiPyApi with the user PKI cert & key using nipyapi.security.set_service_ssl_context
• attempt to connect with NiFi

To fix/workaround, I am instead:

        ssl_context: SSLContext = create_default_context(Purpose.SERVER_AUTH)
        ssl_context.load_cert_chain(
                certfile=cert,
                keyfile=key,
                password=key_password
        )
        ssl_context.load_verify_locations(cafile=ca)
        ssl_context.check_hostname = True
        nipyapi.config.nifi_config.ssl_context = ssl_context

        nipyapi_config.verify_ssl = True

I assume it would be the same for NiFi Registry, although I haven't tested because I've been able to do everything I needed with NiFi Toolkit instead of NiPyApi.

Additional Note

If running NiFi with the "Single User" setup, and not having cert/key files available (they could probably be extracted from the auto-generated Keystore & Truststore, but I haven't), I've instead setup an unverified SSL Context:

        nipyapi.config.nifi_config.ssl_context = ssl._create_unverified_context()
        nipyapi_config.verify_ssl = False
        disable_warnings(InsecureRequestWarning)

Urgency

Not urgent as I figured out a workaround.

A suggested approach might be to allow users to specify the Purpose for the SSL Context through the NiPyApi function - this would allow users to be more explicit for their version of Python (if such a background change is in fact what's caught me out here)

[Chaffelson]

Thanks for this detailed write up @ChrisSamo632
I think this highlights a missing set of tests in NiPy, as the current 'secure' test is really TLS/LDAP (a common requirement), but I think we should also be testing mTLS.
I will make a note to implement a docker test setup for mTLS and try to cover this case.