Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add queues for triggerers. #11

Closed
wants to merge 1 commit into from
Closed

Conversation

avkirilishin
Copy link
Owner

@avkirilishin avkirilishin commented Dec 20, 2023

Closes:

Pull Request for discussing the approach. If this approach is acceptable, I will add queue support for more triggers, add tests, update the documentation, and assess the database load under a large number of triggers.

You can check how triggered's queues work with the sample DAG:

from datetime import time
import pendulum

from airflow import DAG
from airflow.operators.empty import EmptyOperator
from airflow.sensors.time_sensor import TimeSensorAsync


with DAG(
    'trigger-dag',
    schedule='@daily',
    start_date=pendulum.DateTime(2023, 12, 19),
) as dag:
    sensor = TimeSensorAsync(task_id="sync_sensor", target_time=time(10, 59))
    sensor_default = TimeSensorAsync(
        task_id="sync_sensor_default", target_time=time(23, 59), trigger_queue="default")
    sensor_q1 = TimeSensorAsync(task_id="sync_sensor_q1", target_time=time(10, 59), trigger_queue="queue1")
    dummy_op = EmptyOperator(task_id="dummy_op")
    sensor >> dummy_op
    sensor_default >> dummy_op
    sensor_q1 >> dummy_op

photo_2023-12-21_02-21-28

image

@avkirilishin avkirilishin force-pushed the 33818_queues_for_triggerers branch from 107599a to 0936709 Compare December 20, 2023 22:16
@avkirilishin avkirilishin force-pushed the 33818_queues_for_triggerers branch from 439c37e to 55f9475 Compare December 20, 2023 22:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant