Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Canvas sandbox dependencies #350

Merged
merged 16 commits into from
May 2, 2024
Merged

Update Canvas sandbox dependencies #350

merged 16 commits into from
May 2, 2024

Conversation

sydneynotthecity
Copy link
Contributor

The sandbox_update DAG broke because we changed the name of the upstream DAG that the workflow was dependent on. This PR addresses the name change and refactors the code to be less brittle.

  • The DAG originally updated data based on current_date(). If the pipeline broke, backfilling missing data required manual intervention. The query is now date parameterized by batch_start_date and will backfill missing days because catchup=True.
  • copy statements pointed to the internal dataset, hubble-261722.crypto_stellar_internal_2. Updated the statements to use crypto-stellar.crypto_stellar.
  • Added the state table DAG as an additional dependency. We missed this

Note: We will probably need to run the sandbox_create DAG because the column order on the public tables differ from internal. It'll be easier to just recreate the tables so that the schemas match.

Future considerations:

  • add data quality checks
  • use a merge statement to ensure duplicate data is not inserted into the table (alternative is to delete/reinsert)
  • utilize table clone feature to copy data instead of manually inserting new data via SQL

@sydneynotthecity sydneynotthecity requested a review from a team as a code owner May 1, 2024 15:29
@sydneynotthecity sydneynotthecity merged commit da65597 into master May 2, 2024
4 checks passed
@sydneynotthecity sydneynotthecity deleted the fix-sandbox branch May 2, 2024 01:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants