Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expose --max-idle-time RQ flag in rqworker management command #688

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

indispeq
Copy link

@indispeq indispeq commented Dec 2, 2024

Django-rq currently does not expose the --max-idle-time flag that python-rq is able to use under the hood to gracefully shut down a worker after a given amount of seconds. Exposing this functionality proves useful in the event when django-rq workers are run within an autoscaling situation where scale-down time cannot be controlled (e.g.: Azure Container Apps).

By running the worker with ./manage.py rqworker --max-idle-time 60 default the worker can gracefully shutdown before it gets killed by autoscaling, avoiding interrupted jobs that fail with an error of:

Work-horse terminated unexpectedly; waitpid returned 15 (signal 15);

Since --max-jobs is already exposed in the rqworker management command, exposing this option is trivial.

Testing process:

  • When the flag is not used, the worker functions as expected.
  • When the flag is used, the worker exited after 60 seconds:
webapp@e15bb6d1e70d$ ./manage.py rqworker --max-idle-time 60 default
13:03:05 Worker rq:worker:05c2a12455804f9d8a3555f5d004837d started with PID 59, version 2.0.0
13:03:05 Subscribing to channel rq:pubsub:05c2a12455804f9d8a3555f5d004837d
13:03:05 *** Listening on default...
13:03:05 Cleaning registries for queue: default
13:04:05 Worker rq:worker:05c2a12455804f9d8a3555f5d004837d: idle for 60 seconds, quitting
13:04:05 Unsubscribing from channel rq:pubsub:05c2a12455804f9d8a3555f5d004837d

Django-rq currently does not expose the --max-idle-time flag that python-rq
is able to use under the hood to gracefully shut down a worker after a given
amount of seconds. Exposing this functionality proves useful in the event when
django-rq workers are run within an autoscaling situation where scale-down time
cannot be controlled (e.g.: Azure Container Apps).

By running the worker with ./manage.py rqworker --max-idle-time 60 default
the worker can gracefully shutdown before it gets killed by autoscaling, avoiding
interrupted jobs that fail with an error of:

Work-horse terminated unexpectedly; waitpid returned 15 (signal 15);

Since --max-jobs is already exposed in the rqworker management command,
exposing this option is trivial.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant