Offload TLS negotiation to I/O threads #1338
Conversation
Signed-off-by: Uri Yagelnik <[email protected]>
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## unstable #1338 +/- ##
============================================
+ Coverage 70.71% 70.74% +0.02%
============================================
Files 115 116 +1
Lines 63159 63307 +148
============================================
+ Hits 44666 44786 +120
- Misses 18493 18521 +28
|
There was a problem hiding this comment.
This implementation basically breaks the connection abstraction, since it has the TLS implementation calling functions related to IO threading (which is supposed to be agnostic to the connection type). I was sort of expecting we would offload the event handler.
src/io_threads.c
Outdated
| @@ -9,6 +9,7 @@ | |||
| static __thread int thread_id = 0; /* Thread local var */ | |||
| static pthread_t io_threads[IO_THREADS_MAX_NUM] = {0}; | |||
| static pthread_mutex_t io_threads_mutex[IO_THREADS_MAX_NUM]; | |||
| void (*tls_negotiation_cb)(void *); | |||
There was a problem hiding this comment.
Can we make a typdef for this, it'll be a bit more readable.
src/connection.h
Outdated
| @@ -56,6 +56,7 @@ typedef enum { | |||
|
|
|||
| #define CONN_FLAG_CLOSE_SCHEDULED (1 << 0) /* Closed scheduled by a handler */ | |||
| #define CONN_FLAG_WRITE_BARRIER (1 << 1) /* Write barrier requested */ | |||
| #define CONN_FLAG_CLIENT (1 << 2) /* Connection is of a client - not a cluster link. */ | |||
There was a problem hiding this comment.
| #define CONN_FLAG_CLIENT (1 << 2) /* Connection is of a client - not a cluster link. */ | |
| #define CONN_FLAG_NO_OFFLOAD (1 << 2) /* Connection should not be offloaded to IO threads. */ |
The goal was to keep the connection layer independent of the the server layer.
src/io_threads.c
Outdated
| if (server.io_threads_num <= 1) { | ||
| return C_ERR; | ||
| } | ||
|
|
||
| if (!(conn->flags & CONN_FLAG_CLIENT)) { | ||
| return C_ERR; | ||
| } | ||
|
|
||
| client *c = connGetPrivateData(conn); | ||
| if (c->io_read_state != CLIENT_IDLE) { | ||
| return C_OK; | ||
| } | ||
|
|
||
| if (server.active_io_threads_num <= 1) { | ||
| return C_ERR; | ||
| } |
There was a problem hiding this comment.
| if (server.io_threads_num <= 1) { | |
| return C_ERR; | |
| } | |
| if (!(conn->flags & CONN_FLAG_CLIENT)) { | |
| return C_ERR; | |
| } | |
| client *c = connGetPrivateData(conn); | |
| if (c->io_read_state != CLIENT_IDLE) { | |
| return C_OK; | |
| } | |
| if (server.active_io_threads_num <= 1) { | |
| return C_ERR; | |
| } | |
| if (server.active_io_threads_num <= 1) { | |
| return C_ERR; | |
| } | |
| if (!(conn->flags & CONN_FLAG_CLIENT)) { | |
| return C_ERR; | |
| } | |
| client *c = connGetPrivateData(conn); | |
| if (c->io_read_state != CLIENT_IDLE) { | |
| return C_OK; | |
| } |
There was a problem hiding this comment.
It seems everywhere else we just check for active threads.
There was a problem hiding this comment.
Consider a scenario where the main thread sends a job to the IO thread. After the IO thread completes the job, the main thread deactivates all threads. In this case, we want to ensure the main thread processes the returned job from the IO thread before performing an accept operation even if the io threads are not active.
To achieve this:
First, check if IO threads are enabled at all (less expensive check)
Then, verify the read_state is not idle
Finally, check the active state
Unlike read/write where we anyway check for the read/write states, the accept flow operates at the connection layer rather than the client layer, so we can't check the read state there.
Signed-off-by: Uri Yagelnik <[email protected]>
We still don't check in any point for the connection type, since the TLS code calls the IO threads to offload the negotiation with a supplied callback, not the other way around. Maybe we can rename it to 'accept' instead of 'tls_negotiate' to be more abstract.
Not sure I get this, would you please elaborate. |
TLS Negotiation Offloading to I/O Threads
Overview
This PR introduces the ability to offload TLS handshake negotiations to I/O threads, significantly improving performance under high TLS connection loads.
Key Changes
Performance Impact
Testing with 650 clients with SET commands and 160 new TLS connections per second in the background:
Throughput Impact of new TLS connections
New Connection Rate
Implementation Details
Main Thread:
I/O Thread:
Related issue:#761