HUBBLE 444 - Refactor Elementary monitoring to run every 30 min #379
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
chowbao
reviewed
Jun 11, 2024
dags/dbt_data_quality_alerts_dag.py
Outdated
| default_args=get_default_dag_args(), | ||
| start_date=datetime(2024, 6, 11, 0, 0), | ||
| description="This DAG runs dbt tests and Elementary alerts at a half-hourly cadence", | ||
| schedule_interval="*/30 * * * *", |
There was a problem hiding this comment.
Can we change the interval to */15,*/45 * * * *?
There was a problem hiding this comment.
Sure, it makes much more sense to run in-between the dbt dags.
chowbao
reviewed
Jun 11, 2024
| task_name = f"{task_name}_with_exclude" | ||
| args.append("--exclude") | ||
| if isinstance(excluded, list): | ||
| args.append(",".join(excluded)) |
There was a problem hiding this comment.
Is this supposed to be a comma or a space? I think the comma means it is the intersection of the items in the excluded list whereas space means both are excluded
There was a problem hiding this comment.
You're right, it should be space-separated to provide union for the arguments; I'm fixing it in the next commit.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR Checklist
PR Structure
otherwise).
Thoroughness
What
The
elementary_slack_alert_dbt_sdf_martstask was moved into a separate DAG that executes independently of the dbt models after some data quality tests, as defined in Alerting for dbt . This way, we will detect and trigger alerts on problems in a timely manner. The DAG is scheduled to run every 30 minutes, which aligns with thedbt_enriched_base_tablesDAG.The dbt tests running in the new DAG use a new tag designed to fit all the unit tests related to the models' data quality.
Why
Elementary monitoring and alerting are currently executed at the very end of the dbt DAGs. If an upstream dbt model fails, this means that elementary never alerts to a data quality issue because the alerting depends on DAG execution status. The result is that alerts are late arriving or arrive after an issue has already been resolved.
Known limitations
This PR must wait for the
singular_testtag to be deployed for the dbt project.