Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Let the stream instance inform how bookmarks and record values for the incremental replication key ought to be compared #2753

Open
edgarrmondragon opened this issue Nov 11, 2024 · 0 comments
Labels
kind/Feature New feature or request valuestream/SDK

Comments

@edgarrmondragon
Copy link
Collaborator

Feature scope

Taps (catalog, state, tests, etc.)

Description

Instead of forcing all values to be JSON-compatible strings and hope we catch all cases, we should allow the stream to define how to the values from the state and the record should be pre-processed:

old_rk_value = to_json_compatible(progress_dict.get("replication_key_value"))
new_rk_value = to_json_compatible(latest_record[replication_key])
if new_rk_value is None:
logger.warning("New replication value is null")
return
if old_rk_value is None or not check_sorted or new_rk_value >= old_rk_value:
progress_dict["replication_key"] = replication_key
progress_dict["replication_key_value"] = new_rk_value
return

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/Feature New feature or request valuestream/SDK
Projects
None yet
Development

No branches or pull requests

1 participant