-
Notifications
You must be signed in to change notification settings - Fork 19
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unify dbt sdf marts in one dbt task #301
Conversation
As discussed in the meeting I think we would like to keep each tag running in a separate task. The elementary issue was more pronounced because we were running everything as a separate task (stg -> conform -> mart etc...) so there were tons of tasks finishing and writing to elementary at the same time causing the error. Where did you see this causing the elementary issue? Do you have link to an example in airflow? |
Before I had not seen this error anywhere except in the ticket you had opened, I created this PR because I realized that this sdf_marts Dag could suffer from this problem due to the separate tasks. However, today while I was creating a task for Airflow that sends alerts from elementary to slack I received this error while running Dag sdf_marts. |
I think the frequency of the issues is low enough where it should be fine for it to happen and we just let the task retry. If we condense all models to be run at the same time it makes debugging much harder. I would say we can probably close this pr/issue |
This PR adds an if statement that allows you to insert more than one tag for the
dbt_task
function, with this we can run all the tags in the same run to prevent errors about concurrent updates mentioned in the HUBBLE-301 ticket.