Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Switch from environment variable to behavior flag for gating microbatch functionality #323

Merged
merged 7 commits into from
Nov 8, 2024
Merged

Switch from environment variable to behavior flag for gating microbatch functionality #323

merged 7 commits into from
Nov 8, 2024

Conversation

QMalcolm
Copy link
Contributor

@QMalcolm QMalcolm commented Oct 1, 2024 •
edited
Loading

resolves #327

Problem

We're currently gating microbatch functionality via environment variables, and we should instead use a behavior flag

Solution

Switch to behavior flag for gating microbatch functionality

Checklist

  • I have read the contributing guide and understand what's expected of me
  • I have run this code in development, and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc.) or this PR has already received feedback and approval from Product or DX

@cla-bot cla-bot bot added the cla:yes label Oct 1, 2024
This was referenced Oct 1, 2024
Merged
Open
Closed
@QMalcolm QMalcolm marked this pull request as ready for review October 1, 2024 21:55
@QMalcolm QMalcolm requested a review from a team as a code owner October 1, 2024 21:55
@QMalcolm QMalcolm mentioned this pull request Oct 21, 2024
Closed
4 tasks
@QMalcolm
Copy link
Contributor Author

QMalcolm commented Nov 7, 2024

So it turns out, we actually need to do MUCH less. I'm going to drop a bunch of commits... Like half of them. And then also squash some of them. And then make some changes to simplify the tests further.

mikealfare
mikealfare reviewed Nov 8, 2024
View reviewed changes
Copy link
Contributor

@mikealfare mikealfare left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One recommendation that I think solves the "always warning" issue.

dbt/adapters/base/impl.py Outdated Show resolved Hide resolved
dbt/adapters/base/impl.py Show resolved Hide resolved
@QMalcolm
Add .no_warn to the evaluation of `require_batched_execution_for_cu…
2f436ba 
…stom_microbatch_strategy`

The situation in which we want to warn the user about the behavior flag is
exceedingly small. Specifically, we want to fire a deprecation warning when
1. The behavior flag is false
2. The user is executing a `microbatch` model
3. The user has a custome `microbatch` macro

In dbt-adapters we could check (1) and (2), but we have no ability to
check (3). Thus emitting the warning in dbt-adapters upon evaluation of
the behavior flag `require_batched_execution_for_custom_microbatch_strategy`
is _undesirable_. Instead we'll be emitting a deprecation warning from
dbt-core.
@QMalcolm
Revert "Revert/drop: Temporarily bump versions"
c151fd6 
This reverts commit 2b42247.
@QMalcolm QMalcolm merged commit 9c08dea into main Nov 8, 2024
14 checks passed
@QMalcolm QMalcolm deleted the microbatch-behavior-flag branch November 8, 2024 22:25
MichelleArk
MichelleArk reviewed Nov 8, 2024
View reviewed changes
dbt-tests-adapter/dbt/tests/adapter/incremental/test_incremental_microbatch.py
@@ -63,7 +61,6 @@ def assert_row_count(self, project, relation_name: str, expected_row_count: int)

assert len(result) == expected_row_count, f"{relation_name}:{pformat(result)}"

@mock.patch.dict(os.environ, {"DBT_EXPERIMENTAL_MICROBATCH": "True"})
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Let's make sure to apply this change to concrete adapters, I think it's just BigQuery. The existing tests shouldn't break if we left it in, so it's just mostly for cleanliness:

(feel free to start a fresh dbt-bigquery PR if you'd like!)

@MichelleArk
Copy link
Contributor

MichelleArk commented Nov 8, 2024 •
edited
Loading

Release planning: we'll need (patch?) releases of dbt-tests-adapter as well as dbt-adapters for these changes. Given microbatch is in beta status, it feels alright to just do patches here.

Edit (thinking out loud more here)
The two cases are:

  • we're removing support for the env variable opt-in, but microbatch builtin behaviour will continue to work for any projects that have made use of the removed env var.
  • if the env var was not set, we are now adding 'microbatch' to builtins, which can be opted out of with the new behaviour flag.

What do you think?

@QMalcolm
Copy link
Contributor Author

QMalcolm commented Nov 8, 2024

Release planning: we'll need (patch?) releases of dbt-tests-adapter as well as dbt-adapters for these changes. Given microbatch is in beta status, it feels alright to just do patches here. What do you think?

That feels right to me. @mikealfare Does that sound right?

@MichelleArk MichelleArk mentioned this pull request Nov 12, 2024
Merged
4 tasks
@QMalcolm QMalcolm mentioned this pull request Nov 13, 2024
Closed
1 task
This was referenced Nov 13, 2024
Closed
Merged
@colin-rogers-dbt colin-rogers-dbt mentioned this pull request Dec 4, 2024
Draft
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@MichelleArk MichelleArk MichelleArk left review comments

@mikealfare mikealfare mikealfare approved these changes

Assignees
No one assigned
Labels
cla:yes
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Microbatch] Use behavior flag to gate microbatch functionality
3 participants
@QMalcolm @MichelleArk @mikealfare