Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Deprecate amazon async provider #1410

Closed
19 tasks done
Lee-W opened this issue Jan 2, 2024 · 15 comments · Fixed by #1480
Closed
19 tasks done

Deprecate amazon async provider #1410

Lee-W opened this issue Jan 2, 2024 · 15 comments · Fixed by #1480
Assignees
@Lee-W
Labels
feature New feature or request
Milestone
1.19.0

Comments

@Lee-W
Copy link
Contributor

Lee-W commented Jan 2, 2024 •
edited
Loading

Is your feature request related to a problem? Please describe.

as most of the functionality has been contributed back to OSS airflow, we're going to leverage the code there and deprecate this project

Describe the solution you'd like

  1. Contribute the delta to the OSS provider
  2. Point the astronomer provider to the OSS provider
  3. Do the release and wait for a month or so before we remove the astronomer provider in a major release

Describe alternatives you've considered

Additional context

Current Progress

  • Number of operator changes on OSS airflow needed: 10 (finished: 7)
  • Number of operators to deprecate: 18 (finished: 9)
  1. EmrContainerSensorAsync
  2. EmrStepSensorAsync
  3. EmrJobFlowSensorAsync
  4. RedshiftClusterSensorAsync
  5. BatchSensorAsync
  6. S3KeySensorAsync
  7. S3KeySizeSensorAsync
    • Note: already deprecated
    • Action: -
    • Deprecation PR: -
    • OSS Airflow PR: -
    • Minimum Provider Version: -
  8. S3KeysUnchangedSensorAsync
  9. S3PrefixSensorAsync
    • Note: already deprecated
    • Action: -
    • Deprecation PR: -
    • OSS Airflow PR: -
    • Minimum Provider Version: -
  10. EmrContainerOperatorAsync
  11. RedshiftDeleteClusterOperatorAsync
  12. RedshiftResumeClusterOperatorAsync
  13. RedshiftPauseClusterOperatorAsync
  14. BatchOperatorAsync
  15. RedshiftSQLOperatorAsync
    • Note: deprecated in OSS airflow
    • Action: keep it as it is
    • Deprecation PR: -
    • OSS Airflow PR: -
    • Minimum Provider Version: -
  16. SageMakerProcessingOperatorAsync
  17. SageMakerTransformOperatorAsync
  18. SageMakerTrainingOperatorAsync (WIP)
  19. RedshiftDataOperatorAsync
@Lee-W Lee-W added the feature New feature or request label Jan 2, 2024
@Lee-W Lee-W added this to the 2.0.0 milestone Jan 2, 2024
@Lee-W Lee-W self-assigned this Jan 2, 2024
@phanikumv phanikumv changed the title Drepcreate amazon provider Depcrate amazon async provider Jan 2, 2024
@phanikumv phanikumv changed the title Depcrate amazon async provider Deprecate amazon async provider Jan 2, 2024
@phanikumv
Copy link
Collaborator

Refer to apache/airflow#36523

@Lee-W
Copy link
Contributor Author

Lee-W commented Jan 3, 2024

got some push back in apache/airflow#36416 and apache/airflow#36523. already started the discussion with the community. started working on operator we've not yet contributed back instead of missing the check

@Lee-W
Copy link
Contributor Author

Lee-W commented Jan 4, 2024

Add draft apache/airflow#36578 and apache/airflow#36550

@Lee-W
Copy link
Contributor Author

Lee-W commented Jan 4, 2024 •
edited
Loading

Add draft apache/airflow#36586

Different from its original RedshiftDataOperatorAsync implementation, I follow how it's implemented on OSS airflow. The main difference is checking statement_id vs query_ids.

@Lee-W
Copy link
Contributor Author

Lee-W commented Jan 8, 2024

Wrap up apache/airflow#36578 and add apache/airflow#36658

@Lee-W
Copy link
Contributor Author

Lee-W commented Jan 10, 2024 •
edited
Loading

As of Jan 10th

  • Deprecation PR in Review
    • EmrJobFlowSensorAsync
    • S3KeysUnchangedSensorAsync
    • EmrContainerOperatorAsync
  • Wait for upstream review
    • S3KeySensorAsync, S3KeySizeSensorAsync
    • BatchOperatorAsync
    • SageMakerProcessingOperatorAsync
    • SageMakerTransformOperatorAsync
  • Work in progress
    • SageMakerTrainingOperatorAsync
    • RedshiftDataOperatorAsync, RedshiftSQLOperatorAsync
  • Wait for the upstream release
    • RedshiftClusterSensorAsync
    • RedshiftDeleteClusterOperatorAsync, RedshiftResumeClusterOperatorAsync, RedshiftPauseClusterOperatorAsync

@Lee-W
Copy link
Contributor Author

Lee-W commented Jan 11, 2024

As of Jan 11th

  • Deprecation PR in Review
    • EmrJobFlowSensorAsync
    • RedshiftDeleteClusterOperatorAsync, RedshiftResumeClusterOperatorAsync, RedshiftPauseClusterOperatorAsync
  • Work in progress
    • SageMakerTrainingOperatorAsync
    • RedshiftDataOperatorAsync, RedshiftSQLOperatorAsync
  • Wait for the upstream release
    • RedshiftClusterSensorAsync
    • S3KeySensorAsync, S3KeySizeSensorAsync
    • BatchOperatorAsync
    • SageMakerProcessingOperatorAsync
    • SageMakerTransformOperatorAsync

@Lee-W
Copy link
Contributor Author

Lee-W commented Jan 15, 2024

As of Jan 15th

  • Deprecation PR in Review
    • EmrJobFlowSensorAsync
    • RedshiftDeleteClusterOperatorAsync, RedshiftResumeClusterOperatorAsync, RedshiftPauseClusterOperatorAsync
  • Wait for upstream review
    • SageMakerTrainingOperatorAsync
  • Work in progress
    • RedshiftDataOperatorAsync, RedshiftSQLOperatorAsync
  • Wait for the upstream release
    • RedshiftClusterSensorAsync
    • S3KeySensorAsync, S3KeySizeSensorAsync
    • BatchOperatorAsync
    • SageMakerProcessingOperatorAsync
    • SageMakerTransformOperatorAsync

@Lee-W
Copy link
Contributor Author

Lee-W commented Jan 16, 2024 •
edited
Loading

As of Jan 16th

@Lee-W
Copy link
Contributor Author

Lee-W commented Jan 17, 2024

As of Jan 17th

@Lee-W
Copy link
Contributor Author

Lee-W commented Jan 19, 2024

As of Jan 19th

  • Wait for upstream review
  • Wait for the upstream release
    • RedshiftDataOperatorAsync, RedshiftSQLOperatorAsync
    • RedshiftClusterSensorAsync
    • S3KeySensorAsync, S3KeySizeSensorAsync
    • BatchOperatorAsync
    • SageMakerProcessingOperatorAsync
    • SageMakerTransformOperatorAsync

@Lee-W Lee-W modified the milestones: 2.0.0, 1.19.0 Jan 19, 2024
@Lee-W
Copy link
Contributor Author

Lee-W commented Jan 24, 2024 •
edited
Loading

As of Jan 24th

@Lee-W
Copy link
Contributor Author

Lee-W commented Jan 24, 2024

After working on RedshiftSQLOperatorAsync a bit, I think we should not deprecate it and keep it as it is just like the snowflake operator instead.

@Lee-W
Copy link
Contributor Author

Lee-W commented Jan 29, 2024 •
edited
Loading

As of Jan 29th

@phanikumv
Copy link
Collaborator

Waiting for OSS provider release

@pankajkoti pankajkoti mentioned this issue Feb 18, 2024
Merged
@Lee-W Lee-W closed this as completed Feb 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Lee-W Lee-W

Labels
feature New feature or request
Projects
None yet
Milestone
1.19.0
Development

Successfully merging a pull request may close this issue.

Update upstream providers versions from RC versions astronomer/astronomer-providers
2 participants
@Lee-W @phanikumv