Intermittent timeout during service certificate renewal transaction

[achamayou]

Test failure in https://dev.azure.com/MSRC-CCF/CCF/_build/results?buildId=87801&view=logs&j=b6a3fffa-f95e-580b-9bcb-4ae7d4681578&t=7439823f-f6a2-5514-9be9-467f4c602fdd

59: 2024-10-29 08:13:34.454 | INFO     | {session_coseauth} infra.network:refresh_service_identity_file:1661 - After refresh, service_cert.pem is sha256:dc79c2b7b76ab0458eb84e096b8f443fdb1c40e86c71b9ee79ff66641bec30ce
59: 2024-10-29 08:13:34.477 | INFO     | {session_coseauth} __main__:test_service_cert_renewal_extended:529 - [2] GET /node/network/nodes
59: 2024-10-29 08:13:34.477 | INFO     | {session_coseauth} __main__:test_service_cert_renewal_extended:529 - 200 @4.208 {"nodes":[{"last_written":115,"node_data":{"location":"Secret Base","my_id":"0xdeadbeef","previous_locations":["The Moon"]},"node_id":"f94d005900334e485139e51fb2f33d8291e000d251889f5c9a6b6f8ffe0a7d81","primary":false,"rpc_interfaces":{"primary_rpc_interfac + 3829 chars
59: 2024-10-29 08:13:34.477 | INFO     | {session_coseauth} __main__:test_service_cert_renewal_extended:519 - Renewing service certificate for 77 days
59: 2024-10-29 08:13:34.497 | DEBUG    | {session_coseauth} infra.consortium:make_proposal:186 - Writing proposal to /ccfci/workspace_20241029.7/session_coseauth_governance_test_common/set_service_certificate_validity_proposal.json
59: 2024-10-29 08:13:34.497 | DEBUG    | {session_coseauth} infra.consortium:make_proposal:190 - Writing ballot to /ccfci/workspace_20241029.7/session_coseauth_governance_test_common/set_service_certificate_validity_vote_for.json
59: 2024-10-29 08:13:34.498 | INFO     | {session_coseauth} infra.clients:advance:55 - Advancing clock by 1 seconds
59: 2024-10-29 08:13:48.673 | INFO     | {session_coseauth} infra.clients:post:1187 - [2|cose=member0] POST /gov/members/proposals:create?api-version=2024-07-01 @/ccfci/workspace_20241029.7/session_coseauth_governance_test_common/set_service_certificate_validity_proposal.json
59: 2024-10-29 08:13:48.674 | INFO     | {session_coseauth} infra.network:network:1752 - Stopping network
59: 2024-10-29 08:13:48.675 | INFO     | {session_coseauth} infra.network:stop_all_nodes:866 - All nodes stopped
59: 2024-10-29 08:13:48.894 | ERROR    | {session_coseauth} infra.runner:log_exception:193 - Failure in session_coseauth: CCFConnectionException('Connection still failing after 3s')
59: Traceback (most recent call last):
59: 
59:   File "/CCF/build/env/lib/python3.8/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
59:     yield
59:   File "/CCF/build/env/lib/python3.8/site-packages/httpcore/backends/sync.py", line 62, in start_tls
59:     raise exc
59:   File "/CCF/build/env/lib/python3.8/site-packages/httpcore/backends/sync.py", line 57, in start_tls
59:     sock = ssl_context.wrap_socket(
59:   File "/usr/lib/python3.8/ssl.py", line 500, in wrap_socket
59:     return self.sslsocket_class._create(
59:   File "/usr/lib/python3.8/ssl.py", line 1069, in _create
59:     self.do_handshake()
59:     └ <ssl.SSLSocket [closed] fd=-1, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6>
59:   File "/usr/lib/python3.8/ssl.py", line 1338, in do_handshake
59:     self._sslobj.do_handshake()
59:     └ <ssl.SSLSocket [closed] fd=-1, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6>
59: 
59: socket.timeout: _ssl.c:1128: The handshake operation timed out
59: 
59: 
59: During handling of the above exception, another exception occurred:
59: 
59: 
59: Traceback (most recent call last):
59: 
59:   File "/CCF/build/env/lib/python3.8/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
59:     yield
59:   File "/CCF/build/env/lib/python3.8/site-packages/httpx/_transports/default.py", line 218, in handle_request
59:     resp = self._pool.handle_request(req)
59:            │                         └ <Request [b'POST']>
59:            └ <httpx.HTTPTransport object at 0x7f99a2804280>
59:   File "/CCF/build/env/lib/python3.8/site-packages/httpcore/_sync/connection_pool.py", line 253, in handle_request
59:     raise exc
59:   File "/CCF/build/env/lib/python3.8/site-packages/httpcore/_sync/connection_pool.py", line 237, in handle_request
59:     response = connection.handle_request(request)
59:                │                         └ <Request [b'POST']>
59:                └ <HTTPConnection [CONNECTION FAILED]>
59:   File "/CCF/build/env/lib/python3.8/site-packages/httpcore/_sync/connection.py", line 86, in handle_request
59:     raise exc
59:   File "/CCF/build/env/lib/python3.8/site-packages/httpcore/_sync/connection.py", line 63, in handle_request
59:     stream = self._connect(request)
59:              │             └ <Request [b'POST']>
59:              └ <HTTPConnection [CONNECTION FAILED]>
59:   File "/CCF/build/env/lib/python3.8/site-packages/httpcore/_sync/connection.py", line 150, in _connect
59:     stream = stream.start_tls(**kwargs)
59:     │        │                  └ {'ssl_context': <ssl.SSLContext object at 0x7f99a28074c0>, 'server_hostname': '127.118.228.108', 'timeout': 10}
59:     │        └ <httpcore.backends.sync.SyncStream object at 0x7f99a2804700>
59:     └ <httpcore.backends.sync.SyncStream object at 0x7f99a2804700>
59:   File "/CCF/build/env/lib/python3.8/site-packages/httpcore/backends/sync.py", line 62, in start_tls
59:     raise exc
59:   File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
59:     self.gen.throw(type, value, traceback)
59:     │              │     │      └ <traceback object at 0x7f99a28aa940>
59:     │              │     └ timeout('_ssl.c:1128: The handshake operation timed out')
59:     │              └ <class 'socket.timeout'>
59:     └ <contextlib._GeneratorContextManager object at 0x7f99a2804d90>
59:   File "/CCF/build/env/lib/python3.8/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
59:     raise to_exc(exc)
59:           └ <class 'httpcore.ConnectTimeout'>

This test was expanded recently: #6578