Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unify dbt sdf marts in one dbt task #301

Closed
wants to merge 8 commits into from
Closed

Unify dbt sdf marts in one dbt task #301

wants to merge 8 commits into from

Conversation

cayod
Copy link
Contributor

@cayod cayod commented Feb 20, 2024

This PR adds an if statement that allows you to insert more than one tag for the dbt_task function, with this we can run all the tags in the same run to prevent errors about concurrent updates mentioned in the HUBBLE-301 ticket.

@cayod cayod requested a review from a team as a code owner February 20, 2024 20:29
@chowbao
Copy link
Contributor

chowbao commented Feb 21, 2024

As discussed in the meeting I think we would like to keep each tag running in a separate task. The elementary issue was more pronounced because we were running everything as a separate task (stg -> conform -> mart etc...) so there were tons of tasks finishing and writing to elementary at the same time causing the error.

Where did you see this causing the elementary issue? Do you have link to an example in airflow?

@cayod
Copy link
Contributor Author

cayod commented Feb 22, 2024

As discussed in the meeting I think we would like to keep each tag running in a separate task. The elementary issue was more pronounced because we were running everything as a separate task (stg -> conform -> mart etc...) so there were tons of tasks finishing and writing to elementary at the same time causing the error.

Where did you see this causing the elementary issue? Do you have link to an example in airflow?

Before I had not seen this error anywhere except in the ticket you had opened, I created this PR because I realized that this sdf_marts Dag could suffer from this problem due to the separate tasks. However, today while I was creating a task for Airflow that sends alerts from elementary to slack I received this error while running Dag sdf_marts.
look at the first attempt at this task.
https://he02f27a661269b05p-tp.appspot.com/log?dag_id=dbt_sdf_marts&task_id=dbt_build_partnership_assets&execution_date=2024-02-21T16%3A00%3A00%2B00%3A00

@chowbao
Copy link
Contributor

chowbao commented Feb 22, 2024

Before I had not seen this error anywhere except in the ticket you had opened, I created this PR because I realized that this sdf_marts Dag could suffer from this problem due to the separate tasks. However, today while I was creating a task for Airflow that sends alerts from elementary to slack I received this error while running Dag sdf_marts. look at the first attempt at this task. https://he02f27a661269b05p-tp.appspot.com/log?dag_id=dbt_sdf_marts&task_id=dbt_build_partnership_assets&execution_date=2024-02-21T16%3A00%3A00%2B00%3A00

I think the frequency of the issues is low enough where it should be fine for it to happen and we just let the task retry. If we condense all models to be run at the same time it makes debugging much harder.
The other option is to run the tags serially instead of in parallel but I'm not sure that's necessary either.

I would say we can probably close this pr/issue

@cayod cayod closed this Mar 19, 2024
@cayod cayod deleted the unify-sdf-marts branch March 19, 2024 16:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants