Add Support for Delete Tracking in Incremental Append Mode #45898
nimuriel
started this conversation in
Connector Ideas and Features
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Currently, when using incremental append mode with a cursor field (e.g., updated_at), deleted records are not tracked, leading to a loss of visibility on data deletions. I propose introducing a feature similar to Fivetran's "_fivetran_deleted" mechanism, where deleted records are flagged without requiring changes to the updated_at field. This feature would provide an option to track hard deletes by marking records, using the primary key, after deletion.
Existing alternatives, such as modifying the database to track soft deletes or using CDC (Change Data Capture), are either not feasible for large datasets or lead to an overflow of unnecessary records in the data warehouse, especially for high-change-rate tables. A simple delete-tracking mechanism in incremental append mode would address these issues without the need for full CDC or extensive database modifications.
Beta Was this translation helpful? Give feedback.
All reactions