Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RedisBucket raising SynchronousOnlyOperation in Django #186

Open
Hugo-C opened this issue Nov 18, 2024 · 3 comments
Open

RedisBucket raising SynchronousOnlyOperation in Django #186

Hugo-C opened this issue Nov 18, 2024 · 3 comments

Comments

@Hugo-C
Copy link

Hugo-C commented Nov 18, 2024

Hello, I am having trouble using this library with Django.
Indeed using in a Django view a snippet like:

rates = [Rate(1, Duration.SECOND)]
bucket_key = "some key"
redis_client = redis.Redis(host=REDIS_HOSTNAME, port=6379, db=3, password=REDIS_PASSWORD)
bucket = RedisBucket.init(rates, redis_client, bucket_key)
limiter = Limiter(bucket, max_delay=100)

and then doing an ORM request raise an SynchronousOnlyOperation error. It seems to be raised because an event loop was created.
To me the problem is with pyrate_limiter.abstracts.bucket.Leaker.run that create the problematic event loop. I appreciate calling an async code under the hood allows to regroup code with async context, but it seems to also break the compatibility with Django, isn't it ?

@Hugo-C Hugo-C changed the title RedisBucket raising SynchronousOnlyOperation on Django RedisBucket raising SynchronousOnlyOperation in Django Nov 18, 2024
@vutran1710
Copy link
Owner

Probably, its been a long time since I last work with django so things might need some adjustment.
I can do something about it in the free time - but to be honest there will be none for me until next month.
So i guess you have to wait for it now or find a work-around yourself ☹️

@Hugo-C
Copy link
Author

Hugo-C commented Nov 19, 2024

Thanks for the quick reply. Using release 3.2.1 worked fine for me, so it's a good work-around.
I'll try to come up with a PR, that'll at least contains a good test.
Take care.

@Hugo-C
Copy link
Author

Hugo-C commented Nov 23, 2024

I had in mind this test:

from asyncio import get_running_loop
from pyrate_limiter import Duration, Rate, InMemoryBucket, Limiter

rates = [Rate(2, Duration.SECOND)]
bucket = InMemoryBucket(rates)
limiter = Limiter(bucket, max_delay=Duration.SECOND)
limiter.try_acquire("some item")

try:
    get_running_loop()
except RuntimeError:
    pass
else:
    raise RuntimeError("an event loop was created")

As this is what Django does in the end. But this isn't failing for reasons unknown to me. Same behavior with a Redis bucket.
On our side we also had problems with Celery that do not remove running leak background threads. This has resulted in OSError [Errno 24] Too many open files crashes. So background jobs (either based on plain thread or async) are not an option.
So in the end we simply created a custom BucketFactory that do not start background jobs. Instead it only does the cleanup at the start and/or end. More or less like: https://gist.github.com/Hugo-C/100fceff1bba78af63bf1e8f26be02f7.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants