Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Use dbt state to avoid large table rebuilds #628

Merged
merged 1 commit into from
Feb 29, 2024

Conversation

bmtcril
Copy link
Contributor

@bmtcril bmtcril commented Feb 28, 2024

This is a work in progress to avoid rebuilding our large materialized views when we can avoid it. This dbt feature is complicated and requires storing the state of the last run so it can decide what's changed.

Closes: #546

Open questions:

  • How to work around this option being automatically added for commands that it doesn't make sense for (maybe just document that it needs to be turned off for them?)
  • Is a k8s PVC the right way to go to store the state there?
  • Are there other ways to surface that this is happening, or is some documentation enough
  • What things should we consider safe to ignore for "changed" state? Tests, probably. Anything else? https://docs.getdbt.com/reference/node-selection/state-comparison-caveats

@bmtcril bmtcril force-pushed the bmtcril/save_dbt_state branch from ac56ac6 to 60520a5 Compare February 28, 2024 17:45
@bmtcril bmtcril requested a review from SoryRawyer February 28, 2024 19:16
@bmtcril bmtcril force-pushed the bmtcril/save_dbt_state branch from 60520a5 to adb5367 Compare February 28, 2024 20:01
@bmtcril bmtcril force-pushed the bmtcril/save_dbt_state branch from adb5367 to a89c47b Compare February 28, 2024 20:56
@bmtcril bmtcril marked this pull request as ready for review February 28, 2024 21:04
@bmtcril bmtcril force-pushed the bmtcril/save_dbt_state branch from a89c47b to e034246 Compare February 28, 2024 21:17
@bmtcril bmtcril merged commit 9486aad into main Feb 29, 2024
11 checks passed
@bmtcril bmtcril deleted the bmtcril/save_dbt_state branch February 29, 2024 19:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Perf: Only recreate materialzed views when necessary
3 participants