Skip to content

Commit

Permalink
Fix worker threads stalling race condition on RESUME command
Browse files Browse the repository at this point in the history
During a RESUME operation, if a 'MySQL_Thread' is bootstrapping
listeners (in `MySQL_Thread::run_BootstrapListener`) and detect that
`MySQL_Threads_Handler::bootstrapping_listeners` is 'false', the thread
prematurely shuts down its own bootstrapping flag from 'mypolls'
(`ProxySQL_Poll::bootstrapping_listeners`). Since this thread wont ever
bootstrap its corresponding listening sockets, the other working threads
will be stalled waiting on it, eventually triggering the watchdog and
crashing the instance.

Since the bootstrapping operation is sequential, it's expected that all
the threads but the one starting their listening sockets are in an
active wait. A sensible delay has been introduced to reduce the overhead
of such wait.
  • Loading branch information
JavierJF committed Nov 22, 2024
1 parent 0f1262f commit 9c56faf
Show file tree
Hide file tree
Showing 3 changed files with 5 additions and 13 deletions.
1 change: 0 additions & 1 deletion include/ProxySQL_Poll.h
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,6 @@ class ProxySQL_Poll {
MySQL_Data_Stream **myds;
unsigned long long *last_recv;
unsigned long long *last_sent;
std::atomic<bool> bootstrapping_listeners;
volatile int pending_listener_add;
volatile int pending_listener_del;
unsigned int poll_timeout;
Expand Down
16 changes: 5 additions & 11 deletions lib/MySQL_Thread.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -3303,15 +3303,11 @@ void MySQL_Thread::run_BootstrapListener() {
if (n) {
poll_listener_add(n);
assert(__sync_bool_compare_and_swap(&mypolls.pending_listener_add,n,0));
} else {
if (GloMTH->bootstrapping_listeners == false) {
// we stop looping
mypolls.bootstrapping_listeners = false;
}
}
#ifdef DEBUG
usleep(5+rand()%10);
#endif
// The delay for the active-wait is a fraction of 'poll_timeout'. Since other
// threads may be waiting on poll for further operations, checks are meaningless
// until that timeout expires (other workers make progress).
usleep(std::min(std::max(mysql_thread___poll_timeout/20, 10000), 40000) + (rand() % 2000));
}
}

Expand Down Expand Up @@ -3432,9 +3428,7 @@ void MySQL_Thread::run() {
#endif // IDLE_THREADS

pthread_mutex_unlock(&thread_mutex);
if (unlikely(mypolls.bootstrapping_listeners == true)) {
run_BootstrapListener();
}
run_BootstrapListener();

// flush mysql log file
GloMyLogger->flush();
Expand Down
1 change: 0 additions & 1 deletion lib/ProxySQL_Poll.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,6 @@ ProxySQL_Poll::ProxySQL_Poll() {
len=0;
pending_listener_add=0;
pending_listener_del=0;
bootstrapping_listeners = true;
size=MIN_POLL_LEN;
fds=(struct pollfd *)malloc(size*sizeof(struct pollfd));
myds=(MySQL_Data_Stream **)malloc(size*sizeof(MySQL_Data_Stream *));
Expand Down

0 comments on commit 9c56faf

Please sign in to comment.