Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SCD staging tables dedup function added #43

Closed
wants to merge 1 commit into from

Conversation

eduardo-nercolini
Copy link
Contributor

Overview

We have identified issues with the intermediate scd_claimable_balances and scd_liquidity_pools models creation and testing due to duplicate data coming from the sources. To temporarily mitigate the problem while it is being investigated and a permanent solution is proposed, we have implemented a dedup function in the staging models that this tables pull from. This is to prevent downstream model creation and testing failures stemming from this issue.

Key changes

  • row_number window function and qualify function added to stg_claimable_balances and stg_liquidity_pools to remove duplicated data to downstream models.

Other notes

Related links

Checklist

The following items are not covered by automated testing.

  • [N/A] All added/modified models and columns are documented and tested in schema.yml files.
  • All tests pass. OR
    Check one:
    • There are no new test failures, whether or not the existing model was intentionally changed for this PR.
    • This PR causes new test failures and there is a plan in place for addressing them.

@sydneynotthecity sydneynotthecity deleted the feature/scd_dedup_hotfix branch November 14, 2024 14:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant