Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deprecate BatchSensorAsync #1391

Merged
merged 3 commits into from
Dec 21, 2023
Merged

Deprecate BatchSensorAsync #1391

merged 3 commits into from
Dec 21, 2023

Conversation

Lee-W
Copy link
Contributor

@Lee-W Lee-W commented Dec 20, 2023

What's changed

Deprecate BatchSensorAsync and fallback to BatchSensor with deferrable=True

Why this change

The logic of BatchSensorAsync bas been contributed back to OSS airflow as part of BatchSensor

BatchSensorAsync

def execute(self, context: Context) -> None:
"""Defers trigger class to poll for state of the job run until it reaches a failure or a success state"""
if not poke(self, context):
self.defer(
timeout=timedelta(seconds=self.timeout),
trigger=BatchSensorTrigger(
job_id=self.job_id,
aws_conn_id=self.aws_conn_id,
region_name=self.region_name,
poke_interval=self.poke_interval,
),
method_name="execute_complete",
)
def execute_complete(self, context: Context, event: Dict[str, Any]) -> None:
"""
Callback for when the trigger fires - returns immediately.
Relies on trigger to throw an exception, otherwise it assumes execution was
successful.
"""
if "status" in event and event["status"] == "error":
raise_error_or_skip_exception(self.soft_fail, event["message"])
self.log.info(event["message"])

BatchSensor

https://github.com/apache/airflow/blob/999b70178a1f5d891fd2c88af4831a4ba4c2cbc9/airflow/providers/amazon/aws/sensors/batch.py#L98-L133

Copy link

codecov bot commented Dec 20, 2023

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (6781996) 98.57% compared to head (bdc99ef) 98.56%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1391      +/-   ##
==========================================
- Coverage   98.57%   98.56%   -0.02%     
==========================================
  Files          91       91              
  Lines        5460     5421      -39     
==========================================
- Hits         5382     5343      -39     
  Misses         78       78              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Collaborator

@phanikumv phanikumv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what about the integration DAGs that might be using this class, are we planning to remove them at a later point of time?

@phanikumv
Copy link
Collaborator

Please add description about why you are doing this change

@Lee-W
Copy link
Contributor Author

Lee-W commented Dec 21, 2023

what about the integration DAGs that might be using this class, are we planning to remove them at a later point of time?

Yes, I think we should keep it for a while and remove it after we deprecate the whole repo.

Please add description about why you are doing this change

Sure. let me reformat and reword it to make it easier to read.

@Lee-W Lee-W merged commit 37f2d35 into main Dec 21, 2023
11 checks passed
@Lee-W Lee-W deleted the deprecate-batch-sensor branch December 21, 2023 06:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants