Replication Issues Postgres to Snowflake

[luis-settle]

My team is running into replication issues with one of our active streams. We have noticed that there are instances were Airbyte is not picking up all new records from one of our source tables during the sync. The Sync Mode for this table is Incremental Deduped + history, the cursor field is updated_at, and the Primary Key is a composite of two fields that define the uniqueness of a given record in the table. In one example, we will pick up on a row with updated_at = 3:01am and a row with updated_at = 3:03am, but miss out on a row with updated_at = 3:02am. Once we replicate the 3:03am row, there's no way for us to get the 3:02am row without doing a full reset. Typically, large batches of data for this table get created in a Postgres db each morning, with many updated_at's very close to one another. We attempt to replicate that data every 10 minutes, but our replications around the time of those batches getting created have been missing a portion of the rows. @marcosmarxm

[marcosmarxm]

@prateekmukhedkar this is an user having problems with Postgres replication. fyi

[prateekmukhedkar]

@luis-settle are you on self hosted Airbyte or using Airbyte cloud? Asking if you can either share your workspace/connection link or if self-hosted, the logs from the sync you found this behavior?

[luis-settle]

Hi! Sorry I missed your message from last week. we are self hosting Airbyte. Yes, happy to share the logs. Is there a secure space I can share them? Or an email I can use?

[harrison-sommer]

is there any update here? it seems like i'm having a similar issue @prateekmukhedkar