Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Signals support #13

Open
mister-vio opened this issue Jul 25, 2023 · 1 comment
Open

Signals support #13

mister-vio opened this issue Jul 25, 2023 · 1 comment

Comments

@mister-vio
Copy link

mister-vio commented Jul 25, 2023

Hello, is there a way to make async Celery Signals? Do you have an Idea of a workaround?
what I want to achieve is the async initialization of some resources after the worker pool gets created like this:

from celery import signals

@signals.worker_init.connect
async def worker_init_dispatch(*args, **kwargs):
    await print('Do some worker initialization')
@mister-vio
Copy link
Author

mister-vio commented Jul 25, 2023

Found this workaround, the solution is a bit stupid, but it works :) :

import asyncio
import os

from functools import wraps
import celery_aio_pool as aio_pool
from celery import signals

class CustomAsyncIOPool(aio_pool.pool.AsyncIOPool):
    ...

app = Celery('ai_tasks',
                      broker=os.envirion['CELERY_BROKER'],
                      backend=os.envirion['CELERY_BACKEND'],
                      worker_pool=CustomAsyncIOPool)

def async_signal(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        return aio_pool.pool.AsyncIOPool.run_in_pool(func, *args, **kwargs)
    return wrapper

@signals.worker_init.connect
@async_signal
async def worker_init_dispatch(*args, **kwargs):
    await asyncio.sleep(1)
    # Some other async initialization work here
    print('Done')

All other solutions were causing errors like: "attached to a different loop"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant