You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
CeleryEmailBackend ignores the fail_silently option when enqueuing messages in send_messages(). When the Celery message broker is down and DEBUG is False, this causes huge delays in requests where a server error occurs. This allows a DoS attack.
Steps to reproduce:
Ensure Celery is configured to retry failed publishing of tasks (on by default).
Set DEBUG to False in settings.py.
Stop the Celery message broker.
Send a request that causes a server error (you can simply raise an artificial error in a view and then call it).
The response takes about a minute (many times more than the expected time).
Analysis
A typical Django application has several middleware classes in front of the application views. The middleware classes are chained together. To propagate the request, each middleware calls get_response(), which translates to django.core.handlers.exception.convert_exception_to_response.inner(). Its code is:
When an error occurs, it is logged inside response_for_exception(). The default logging configuration attempts to send an error report to the website admins, and passes fail_silently to the email backend. But when the Celery message broker is down, CeleryEmailBackend does not suppress the exception thrown by Celery/Kombu (kombu.exceptions.OperationalError), and the exception escapes out of response_for_exception(). It is caught by the get_response() call of the previous middleware, which in turn repeats the process. This retry process is repeated as many times as the number of middleware classes configured, which significantly multiplies the expected response delay.
Note: This is not caused by a Celery issue ((celery/celery#4296)). I am aware of this issue and have applied a workaround, but the problem described here still persists.
Solution
I have managed to work around this issue by wrapping CeleryEmailBackend in the following subclass:
A fix in the library code would be similar and quite simple - wrap calls to send_emails.delay() in a try block, and if an OperationalError exception is caught and fail_silently is set, suppress the exception, possibly logging it to prevent it from being unnoticed.
Overview
CeleryEmailBackend
ignores thefail_silently
option when enqueuing messages insend_messages()
. When the Celery message broker is down andDEBUG
isFalse
, this causes huge delays in requests where a server error occurs. This allows a DoS attack.Steps to reproduce:
DEBUG
toFalse
insettings.py
.Expected behavior
The response should take 1-2 seconds, depending on the configured Celery message sending retry policy (see (https://docs.celeryproject.org/en/latest/userguide/calling.html#calling-retry)).
Actual behavior
The response takes about a minute (many times more than the expected time).
Analysis
A typical Django application has several middleware classes in front of the application views. The middleware classes are chained together. To propagate the request, each middleware calls
get_response()
, which translates todjango.core.handlers.exception.convert_exception_to_response.inner()
. Its code is:When an error occurs, it is logged inside
response_for_exception()
. The default logging configuration attempts to send an error report to the website admins, and passesfail_silently
to the email backend. But when the Celery message broker is down,CeleryEmailBackend
does not suppress the exception thrown by Celery/Kombu (kombu.exceptions.OperationalError
), and the exception escapes out ofresponse_for_exception()
. It is caught by theget_response()
call of the previous middleware, which in turn repeats the process. This retry process is repeated as many times as the number of middleware classes configured, which significantly multiplies the expected response delay.Note: This is not caused by a Celery issue ((celery/celery#4296)). I am aware of this issue and have applied a workaround, but the problem described here still persists.
Solution
I have managed to work around this issue by wrapping
CeleryEmailBackend
in the following subclass:A fix in the library code would be similar and quite simple - wrap calls to
send_emails.delay()
in a try block, and if anOperationalError
exception is caught andfail_silently
is set, suppress the exception, possibly logging it to prevent it from being unnoticed.Stacktrace
The text was updated successfully, but these errors were encountered: