[PRODUCTION] Update production Airflow environment #382
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
* test time task without affinity * delete affinity from repo * dbt_ * testing config related to stellar_marts * documented * test without var dbt_dataset * with dbt_dataset again * dbt_dataset_for_test * comment excluded
* Created alert_sla_miss callback funcion * Defined test value for export task SLA * Defined sla_miss_callback for DAG level parameter * Increased time task timeout value * Added dev Airflow variable for tasks SLA values * Added slack notification logic for the SLA miss callback * Added SLA parameter to the relavant task building functions * Removed SLA miss callback function and related logic * Added refactored logic for sla miss callback function * Added sla miss callback to desired dags * Adjusted callback logic for sentry integration * Added task SLA values for prod Airflow variables * Changed alert message to better distinguish SLA miss from task fail alerts * Update dev tasks sla and timeout values * Update prod tasks sla and timeout values * Added default sla value to dag default args * Added missing SLA param to build task functions * Standardizing logic for fetching SLA param value * Added sla miss callback reference * Remove default SLA from default args * Set missing sla params to tasks * Updated values for sla variables * Updated build_dbt_task timeout value * Changed variable names to match new naming reference --------- Co-authored-by: chowbao <[email protected]>
sydneynotthecity
added a commit
that referenced
this pull request
Jun 13, 2024
* [PRODUCTION] Update production Airflow environment (#370) * requirements updated * sqlfluff * Use the correct gcs bucket (#372) --------- Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> * [PRODUCTION] Update production Airflow environment (#375) * requirements updated * Update READMEs --------- Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> --------- Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> * [PRODUCTION] Update production Airflow environment (#378) * HUBBLE-409 Update/delete airflow variable names (#377) * Updates for testnet-reset (#380) * Feature/add slas (#359) --------- Co-authored-by: chowbao <[email protected]> --------- Co-authored-by: Laysa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> Co-authored-by: Eduardo Alves <[email protected]> * [PRODUCTION] Update production Airflow environment (#382) * HUBBLE-409 Update/delete airflow variable names (#377) * test time task without affinity * delete affinity from repo * dbt_ * testing config related to stellar_marts * documented * test without var dbt_dataset * with dbt_dataset again * dbt_dataset_for_test * comment excluded * Updates for testnet-reset (#380) * Feature/add slas (#359) * Created alert_sla_miss callback funcion * Defined test value for export task SLA * Defined sla_miss_callback for DAG level parameter * Increased time task timeout value * Added dev Airflow variable for tasks SLA values * Added slack notification logic for the SLA miss callback * Added SLA parameter to the relavant task building functions * Removed SLA miss callback function and related logic * Added refactored logic for sla miss callback function * Added sla miss callback to desired dags * Adjusted callback logic for sentry integration * Added task SLA values for prod Airflow variables * Changed alert message to better distinguish SLA miss from task fail alerts * Update dev tasks sla and timeout values * Update prod tasks sla and timeout values * Added default sla value to dag default args * Added missing SLA param to build task functions * Standardizing logic for fetching SLA param value * Added sla miss callback reference * Remove default SLA from default args * Set missing sla params to tasks * Updated values for sla variables * Updated build_dbt_task timeout value * Changed variable names to match new naming reference --------- Co-authored-by: chowbao <[email protected]> * Update stellar-etl image (#381) * Update stellar-etl image (#383) --------- Co-authored-by: Laysa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> Co-authored-by: Eduardo Alves <[email protected]> --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> Co-authored-by: Eduardo Alves <[email protected]>
sydneynotthecity
added a commit
that referenced
this pull request
Jun 13, 2024
* [PRODUCTION] Update production Airflow environment (#370) * requirements updated * sqlfluff * Use the correct gcs bucket (#372) --------- Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> * [PRODUCTION] Update production Airflow environment (#375) * requirements updated * Update READMEs --------- Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> --------- Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> * [PRODUCTION] Update production Airflow environment (#378) * HUBBLE-409 Update/delete airflow variable names (#377) * Updates for testnet-reset (#380) * Feature/add slas (#359) --------- Co-authored-by: chowbao <[email protected]> --------- Co-authored-by: Laysa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> Co-authored-by: Eduardo Alves <[email protected]> * [PRODUCTION] Update production Airflow environment (#382) * HUBBLE-409 Update/delete airflow variable names (#377) * test time task without affinity * delete affinity from repo * dbt_ * testing config related to stellar_marts * documented * test without var dbt_dataset * with dbt_dataset again * dbt_dataset_for_test * comment excluded * Updates for testnet-reset (#380) * Feature/add slas (#359) * Created alert_sla_miss callback funcion * Defined test value for export task SLA * Defined sla_miss_callback for DAG level parameter * Increased time task timeout value * Added dev Airflow variable for tasks SLA values * Added slack notification logic for the SLA miss callback * Added SLA parameter to the relavant task building functions * Removed SLA miss callback function and related logic * Added refactored logic for sla miss callback function * Added sla miss callback to desired dags * Adjusted callback logic for sentry integration * Added task SLA values for prod Airflow variables * Changed alert message to better distinguish SLA miss from task fail alerts * Update dev tasks sla and timeout values * Update prod tasks sla and timeout values * Added default sla value to dag default args * Added missing SLA param to build task functions * Standardizing logic for fetching SLA param value * Added sla miss callback reference * Remove default SLA from default args * Set missing sla params to tasks * Updated values for sla variables * Updated build_dbt_task timeout value * Changed variable names to match new naming reference --------- Co-authored-by: chowbao <[email protected]> * Update stellar-etl image (#381) * Update stellar-etl image (#383) --------- Co-authored-by: Laysa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> Co-authored-by: Eduardo Alves <[email protected]> --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> Co-authored-by: Eduardo Alves <[email protected]>
sydneynotthecity
added a commit
that referenced
this pull request
Jun 17, 2024
* [PRODUCTION] Update production Airflow environment (#370) * requirements updated * sqlfluff * Use the correct gcs bucket (#372) --------- Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> * [PRODUCTION] Update production Airflow environment (#375) * requirements updated * Update READMEs --------- Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> --------- Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> * [PRODUCTION] Update production Airflow environment (#378) * HUBBLE-409 Update/delete airflow variable names (#377) * Updates for testnet-reset (#380) * Feature/add slas (#359) --------- Co-authored-by: chowbao <[email protected]> --------- Co-authored-by: Laysa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> Co-authored-by: Eduardo Alves <[email protected]> * [PRODUCTION] Update production Airflow environment (#382) * HUBBLE-409 Update/delete airflow variable names (#377) * test time task without affinity * delete affinity from repo * dbt_ * testing config related to stellar_marts * documented * test without var dbt_dataset * with dbt_dataset again * dbt_dataset_for_test * comment excluded * Updates for testnet-reset (#380) * Feature/add slas (#359) * Created alert_sla_miss callback funcion * Defined test value for export task SLA * Defined sla_miss_callback for DAG level parameter * Increased time task timeout value * Added dev Airflow variable for tasks SLA values * Added slack notification logic for the SLA miss callback * Added SLA parameter to the relavant task building functions * Removed SLA miss callback function and related logic * Added refactored logic for sla miss callback function * Added sla miss callback to desired dags * Adjusted callback logic for sentry integration * Added task SLA values for prod Airflow variables * Changed alert message to better distinguish SLA miss from task fail alerts * Update dev tasks sla and timeout values * Update prod tasks sla and timeout values * Added default sla value to dag default args * Added missing SLA param to build task functions * Standardizing logic for fetching SLA param value * Added sla miss callback reference * Remove default SLA from default args * Set missing sla params to tasks * Updated values for sla variables * Updated build_dbt_task timeout value * Changed variable names to match new naming reference --------- Co-authored-by: chowbao <[email protected]> * Update stellar-etl image (#381) * Update stellar-etl image (#383) --------- Co-authored-by: Laysa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> Co-authored-by: Eduardo Alves <[email protected]> --------- Co-authored-by: sydneynotthecity <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]>
sydneynotthecity
added a commit
that referenced
this pull request
Jun 17, 2024
* HUBBLE-409 Update/delete airflow variable names (#377) * test time task without affinity * delete affinity from repo * dbt_ * testing config related to stellar_marts * documented * test without var dbt_dataset * with dbt_dataset again * dbt_dataset_for_test * comment excluded * Updates for testnet-reset (#380) * Feature/add slas (#359) * Created alert_sla_miss callback funcion * Defined test value for export task SLA * Defined sla_miss_callback for DAG level parameter * Increased time task timeout value * Added dev Airflow variable for tasks SLA values * Added slack notification logic for the SLA miss callback * Added SLA parameter to the relavant task building functions * Removed SLA miss callback function and related logic * Added refactored logic for sla miss callback function * Added sla miss callback to desired dags * Adjusted callback logic for sentry integration * Added task SLA values for prod Airflow variables * Changed alert message to better distinguish SLA miss from task fail alerts * Update dev tasks sla and timeout values * Update prod tasks sla and timeout values * Added default sla value to dag default args * Added missing SLA param to build task functions * Standardizing logic for fetching SLA param value * Added sla miss callback reference * Remove default SLA from default args * Set missing sla params to tasks * Updated values for sla variables * Updated build_dbt_task timeout value * Changed variable names to match new naming reference --------- Co-authored-by: chowbao <[email protected]> * Update stellar-etl image (#381) * Update stellar-etl image (#383) * Updated SLA values for dbt eho related tasks (#384) * Patch/merge conflict resolution (#386) * [PRODUCTION] Update production Airflow environment (#370) * requirements updated * sqlfluff * Use the correct gcs bucket (#372) --------- Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> * [PRODUCTION] Update production Airflow environment (#375) * requirements updated * Update READMEs --------- Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> --------- Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> * [PRODUCTION] Update production Airflow environment (#378) * HUBBLE-409 Update/delete airflow variable names (#377) * Updates for testnet-reset (#380) * Feature/add slas (#359) --------- Co-authored-by: chowbao <[email protected]> --------- Co-authored-by: Laysa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> Co-authored-by: Eduardo Alves <[email protected]> * [PRODUCTION] Update production Airflow environment (#382) * HUBBLE-409 Update/delete airflow variable names (#377) * test time task without affinity * delete affinity from repo * dbt_ * testing config related to stellar_marts * documented * test without var dbt_dataset * with dbt_dataset again * dbt_dataset_for_test * comment excluded * Updates for testnet-reset (#380) * Feature/add slas (#359) * Created alert_sla_miss callback funcion * Defined test value for export task SLA * Defined sla_miss_callback for DAG level parameter * Increased time task timeout value * Added dev Airflow variable for tasks SLA values * Added slack notification logic for the SLA miss callback * Added SLA parameter to the relavant task building functions * Removed SLA miss callback function and related logic * Added refactored logic for sla miss callback function * Added sla miss callback to desired dags * Adjusted callback logic for sentry integration * Added task SLA values for prod Airflow variables * Changed alert message to better distinguish SLA miss from task fail alerts * Update dev tasks sla and timeout values * Update prod tasks sla and timeout values * Added default sla value to dag default args * Added missing SLA param to build task functions * Standardizing logic for fetching SLA param value * Added sla miss callback reference * Remove default SLA from default args * Set missing sla params to tasks * Updated values for sla variables * Updated build_dbt_task timeout value * Changed variable names to match new naming reference --------- Co-authored-by: chowbao <[email protected]> * Update stellar-etl image (#381) * Update stellar-etl image (#383) --------- Co-authored-by: Laysa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> Co-authored-by: Eduardo Alves <[email protected]> --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> Co-authored-by: Eduardo Alves <[email protected]> * Help (#388) * [PRODUCTION] Update production Airflow environment (#370) * requirements updated * sqlfluff * Use the correct gcs bucket (#372) --------- Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> * [PRODUCTION] Update production Airflow environment (#375) * requirements updated * Update READMEs --------- Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> --------- Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> * [PRODUCTION] Update production Airflow environment (#378) * HUBBLE-409 Update/delete airflow variable names (#377) * Updates for testnet-reset (#380) * Feature/add slas (#359) --------- Co-authored-by: chowbao <[email protected]> --------- Co-authored-by: Laysa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> Co-authored-by: Eduardo Alves <[email protected]> * [PRODUCTION] Update production Airflow environment (#382) * HUBBLE-409 Update/delete airflow variable names (#377) * test time task without affinity * delete affinity from repo * dbt_ * testing config related to stellar_marts * documented * test without var dbt_dataset * with dbt_dataset again * dbt_dataset_for_test * comment excluded * Updates for testnet-reset (#380) * Feature/add slas (#359) * Created alert_sla_miss callback funcion * Defined test value for export task SLA * Defined sla_miss_callback for DAG level parameter * Increased time task timeout value * Added dev Airflow variable for tasks SLA values * Added slack notification logic for the SLA miss callback * Added SLA parameter to the relavant task building functions * Removed SLA miss callback function and related logic * Added refactored logic for sla miss callback function * Added sla miss callback to desired dags * Adjusted callback logic for sentry integration * Added task SLA values for prod Airflow variables * Changed alert message to better distinguish SLA miss from task fail alerts * Update dev tasks sla and timeout values * Update prod tasks sla and timeout values * Added default sla value to dag default args * Added missing SLA param to build task functions * Standardizing logic for fetching SLA param value * Added sla miss callback reference * Remove default SLA from default args * Set missing sla params to tasks * Updated values for sla variables * Updated build_dbt_task timeout value * Changed variable names to match new naming reference --------- Co-authored-by: chowbao <[email protected]> * Update stellar-etl image (#381) * Update stellar-etl image (#383) --------- Co-authored-by: Laysa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> Co-authored-by: Eduardo Alves <[email protected]> --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> Co-authored-by: Eduardo Alves <[email protected]> * Patch/fix release conflicts (#390) * [PRODUCTION] Update production Airflow environment (#370) * requirements updated * sqlfluff * Use the correct gcs bucket (#372) --------- Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> * [PRODUCTION] Update production Airflow environment (#375) * requirements updated * Update READMEs --------- Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> --------- Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> * [PRODUCTION] Update production Airflow environment (#378) * HUBBLE-409 Update/delete airflow variable names (#377) * Updates for testnet-reset (#380) * Feature/add slas (#359) --------- Co-authored-by: chowbao <[email protected]> --------- Co-authored-by: Laysa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> Co-authored-by: Eduardo Alves <[email protected]> * [PRODUCTION] Update production Airflow environment (#382) * HUBBLE-409 Update/delete airflow variable names (#377) * test time task without affinity * delete affinity from repo * dbt_ * testing config related to stellar_marts * documented * test without var dbt_dataset * with dbt_dataset again * dbt_dataset_for_test * comment excluded * Updates for testnet-reset (#380) * Feature/add slas (#359) * Created alert_sla_miss callback funcion * Defined test value for export task SLA * Defined sla_miss_callback for DAG level parameter * Increased time task timeout value * Added dev Airflow variable for tasks SLA values * Added slack notification logic for the SLA miss callback * Added SLA parameter to the relavant task building functions * Removed SLA miss callback function and related logic * Added refactored logic for sla miss callback function * Added sla miss callback to desired dags * Adjusted callback logic for sentry integration * Added task SLA values for prod Airflow variables * Changed alert message to better distinguish SLA miss from task fail alerts * Update dev tasks sla and timeout values * Update prod tasks sla and timeout values * Added default sla value to dag default args * Added missing SLA param to build task functions * Standardizing logic for fetching SLA param value * Added sla miss callback reference * Remove default SLA from default args * Set missing sla params to tasks * Updated values for sla variables * Updated build_dbt_task timeout value * Changed variable names to match new naming reference --------- Co-authored-by: chowbao <[email protected]> * Update stellar-etl image (#381) * Update stellar-etl image (#383) --------- Co-authored-by: Laysa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> Co-authored-by: Eduardo Alves <[email protected]> --------- Co-authored-by: sydneynotthecity <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> --------- Co-authored-by: Laysa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> Co-authored-by: Eduardo Alves <[email protected]> Co-authored-by: sydneynotthecity <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR was auto-generated by GitHub Actions.
After merged and closed, this PR will trigger an action that updates DAGs, libs and schemas files from prod Airflow.