[Research] How to do async tasks

[Uxio0]

Define what to use for async tasks:

• FastAPI background tasks (it shouldn't work, as we need them to be async)
• Celery Beat + Celery
• Dramatiq

Most tasks will be triggered when reading messages from our RabbitMQ event queue

[falvaradorodriguez]

As an initial result of this work, I would like to share some of the conclusions:

FastAPI background tasks would not be a good idea for the kind of processing we need. Even in its own documentation it recommends to use some tool like Celery (https://fastapi.tiangolo.com/tutorial/background-tasks/#caveat)

Between Celery and Dramatiq, I haven't found a strong characteristic that makes me choose one or the other. As main differences I see the following:

• Celery has more life time and has more community behind it (github celery) vs (github dramatiq)

• Celery has greater capacity for adjustment and management. It is possible to make finer grain settings than Dramatiq.

• Celery indicates that its strongest point is that it is scalable and able to effectively manage a big amount of simultaneous tasks. In theory Dramatiq is less scalable, because it is intended to be lighter and consume fewer resources.

• Dramatiq is lighter and less complex than Celery, and theoretically consumes fewer resources during execution.

• In theory Dramatiq and Celery are compatible in terms of the functionalities they offer: https://dramatiq.io/motivation.html#compared-to. To verify this a bit, I have made a proof of concept with FastAPI and Dramatiq, since Celery is a known project in our ecosystem and seems to fit the intended use case (Add proof of async tasks with dramatic and periodiq #10). Note: Dramatiq does not offer cron tasks natively, so I had to use Periodiq in conjunction with Dramatiq to get this functionality.

• Celery would be configurable together with FastAPI, more or less in the same way as Dramatiq. (Example)

• Dramatiq's way of handling asynchrony is closer to FastAPI than Celery's way of handling asynchrony. It is also particularly suitable for I/O operations such as network requests. Something that is close to the use case of this project.

• In terms of opinions of third parties who have used both projects, there is no majority feeling towards one of the two projects. I give two examples that are quite explanatory. (Example) (Example)

Now, as a personal opinion and for discussion with the team: Since we know the "problems" we are having with Celery in other projects, and in view of the proof of concept, maybe we can give a chance to Dramatiq for this project. In this way we can assess whether we avoid some of the known problems and the tool is as light in terms of resource consumption as it claims.

What do you think @safe-global/core-api?