-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Microbatch: event_time ref + source filtering #10594
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #10594 +/- ##
==========================================
- Coverage 88.90% 88.89% -0.02%
==========================================
Files 180 180
Lines 22785 22842 +57
==========================================
+ Hits 20257 20305 +48
- Misses 2528 2537 +9
Flags with carried forward coverage won't be shown. Click here to find out more.
|
…ack + batch_size work
… using those names
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Self approving a PR that I was a primary contributor to 🙈
However, we've had external eyes on it. It's well gated, well tested, and confirmed backwards compatible. LGTM 🚀
echoing this as well ✅ |
Resolves #9490
Resolves #10635
Resolves #10637
Resolves #10638
Resolves #10636
Resolves #10662
Resolves #10639
Details
We're working to introduce a new feature, Microbatch Incremental Models! The goal being that microbatch incremental models will break updates to microbatch models into batched updates for a given time frame. There's more to microbatch incremental models than that, but that's the gist of if.
This PR doesn't actually do the breaking up into batches. However, what this PR does is set up the fundamentals to do so. There are a few new model configs we need:
event_time
,batch_size
,lookback
. Additionally, there are some CLI flags we need:--event-time-start
and--event-time-end
(with associated environment variablesDBT_EVENT_TIME_END
andDBT_EVENT_TIME_START
). This PR makes those available and uses them to set the foundation of generating event time filters and determining when the event time filters should be generated. Finally, all this foundational functionality is gated by a new environment variableDBT_EXPERIMENTAL_MICROBATCH
.Checklist