Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor DataprocCreateBatchOperator and Dataproc system tests #92

Closed
wants to merge 1 commit into from

Conversation

moiseenkov
Copy link
Collaborator

@moiseenkov moiseenkov commented Aug 14, 2024

Refactored DataprocCreateBatchOperator:

  • significantly refactored the execute() method for decreasing its accumulated complexity and code duplication.
  • made the batch_id parameter optional as it is supported by API
  • made the region parameter required because (1) it is required by the API, and (2) it was already required de-facto because the operator used to raise and exception manually: raise AirflowException("Region should be set here")
  • added a specific error message to the operator logs (in both deferrable=True|False modes), so it would be more convenient for users to debug their batch jobs using the operator logs directly.

Also additionally slight refactored Dataproc system tests:

  • reduced parallelism
  • added retry to cluster creation tasks in a hope to suppress the error

This PR also rolls back changes in pre-commit hook made for Dataflow system tests. From now those changes rea not needed.

@moiseenkov moiseenkov force-pushed the dataproc/create_batch_operator/fix_logging branch 3 times, most recently from 9e45eab to 1e7773a Compare August 16, 2024 08:48
@moiseenkov moiseenkov force-pushed the dataproc/create_batch_operator/fix_logging branch from 1e7773a to 0f25e9a Compare August 16, 2024 08:50
@moiseenkov moiseenkov closed this Aug 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants