High CPU usage on database when worker count > perform_limit

[sairam]

I went through this scenario.
good_job version - 3.7.4

Purpose:
Rate limit messages to upstream. (decided to use perform_limit: 4)

Settings:

• Perform_limit: 4 . Job concurrency key: 'constant'
• Worker thread count: 6 (3 instances x 2 threads)
• There were 20,000 messages in the queue

Effects:

• The jobs were getting re-enqueued back into the system with increasing retry count
• the CPU spiked on the database hosting the good_job

Fixed via:
Reduce the number of workers to match the perform_limit.

How Good job or any queue works:

• Workers pull messages and try to perform tasks
• They understand there is a limitation, they put it back with increased retry count

I want to understand if there is a way to throttle this to avoid the high CPU count and a way to control the threads to not go above the perform limit. This is complicated given there can be multiple jobs in a queue and threads are for queues. This scenario happens when there is one worker, one thread, I want to rate limit sending messages to upstream

[bensheldon]

@sairam hmmm. I believe that you're experiencing the Concurrency Control's normal behavior: it will retry-with-incremental-backoff, which can put stress on the database because it's fetching-and-locking-and-rescheduling a lot if there are 20k jobs in the queue.

In your case, I would recommend that you put those jobs onto a single named queue, and then operationally limit the number of processes/threads that are pulling from that queue to be 4.

I'm curious, why do you want to limit perform to only 4 concurrent jobs?