Optimized DAGs parse time

[lucaszanotelli]

Summary

This PR focuses on implementing optimizations recommended by Google Cloud to reduce Airflow DAG parse times, as outlined in their blog article.

Changes

• Reduced Variable.get() in top level code.
• Reorganized DAG structures and dependencies to align with the recommended practices.
• Streamlined the import of external dependencies to enhance DAG parsing efficiency.
• Eliminated redundant imports and organized dependencies based on DAG requirements.

Testing

I tested these modifications in our test environment and the DAG parse time saw a reduction of +60%.

full line: DAG parse time after changes

Next Steps

• Use .airflowignore
• Review paused DAGs

[sydneynotthecity]

Changes look good.

Was the intent to replace all instances of Variable.get() with {{ var.value.x }}? If so, there are a quite a few instances of the original Variable.get() references that remain.

This change is quite exciting--our DAG parse times were really long and this reduction will be a big impact in making Airflow healthier.

[lucaszanotelli]

Was the intent to replace all instances of Variable.get() with {{ var.value.x }}? If so, there are a quite a few instances of the original Variable.get() references that remain.

You can only replace Variable.get() with {{ var.value.x }} when you use the value inside a templateable field.
More info about Airflow templating here: https://docs.astronomer.io/learn/templating

If you're using the value at top-level code (we are) and/or inside non templateable fields (also we are), there isn't a pretty way to access the Airflow variables.

I thought about parsing the variable JSON file that we upload to the bucket but to accomplish that but I'd need to invest more time in this.

[sydneynotthecity]

Alright, that makes sense. I don't think we need to invest more time in it unless you think we would have significant benefit. The current improvement is good enough as a first pass