Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Sentry failure callback to custom python operator #479

Merged
merged 2 commits into from
Aug 19, 2024

Conversation

sydneynotthecity
Copy link
Contributor

PR Checklist

PR Structure

  • This PR has reasonably narrow scope (if not, break it down into smaller PRs).
  • This PR avoids mixing refactoring changes with feature changes (split into two PRs
    otherwise).
  • This PR's title starts with the jira ticket associated with the PR.

Thoroughness

  • This PR adds tests for the most critical parts of the new functionality or fixes.
  • I've updated the README with the added features, breaking changes, new instructions on how to use the repository.

What

Airflow was not passing failed task status to Sentry correctly for del_ins_ tasks in the history_table_export DAG. The team was not getting notified when the task would fail, requiring manual inspection in Airflow to know if such tasks failed.

Why

I think the callback was defined too deep into the del_ins operator. There is an custom Python wrapper operator, called build_del_ins_operator that calls the actual operators that delete and insert data in BigQuery. The callback was set in the delete/insert operators, but not the Python wrapper. Adding to the wrapper to see if errors are sent to Sentry.

Known limitations

Needs testing in test to confirm that it works correctly.

@sydneynotthecity sydneynotthecity requested a review from a team as a code owner August 19, 2024 14:48
@sydneynotthecity sydneynotthecity merged commit b41e514 into master Aug 19, 2024
5 checks passed
@sydneynotthecity sydneynotthecity deleted the fix-sentry-alerts branch November 21, 2024 03:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants