Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add duplicity handle for subscription stream #71

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Add duplicity handle for subscription stream #71

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d2b6a07
Add duplicity handle for subscription stream
butkeraites-hotglue Nov 11, 2024
e6b728a
Adjusts the unicity guarantee mechanism
butkeraites-hotglue Nov 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,4 +11,5 @@ singer-check-tap-data
dist/
__pycache__/
.idea
.secrets
.secrets
.venv
12 changes: 8 additions & 4 deletions tap_chargebee/streams/base.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -243,10 +243,14 @@ def sync_data(self):
if self.ENTITY == 'coupon':
for coupon in Util.coupons:
to_write.append(coupon)
if self.ENTITY == 'transaction':
# store ids to clean dupplicates
to_write = [record for record in to_write if record["id"] not in ids]
ids.extend([trans["id"] for trans in to_write])
if self.ENTITY in ['transaction', 'subscription']:
# store ids to clean dupplicates, keep only the last appearance of a record id
new_records = []
for record in reversed(to_write):
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if the goal is to keep the latest updated record, I think this will only work if the duplicates are in the same response page, if that's not that relevant please ignore.

Copy link
Author

@butkeraites-hotglue butkeraites-hotglue Nov 12, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pushed a fix for that. Thank you 🙏

if record["id"] not in ids:
new_records.append(record)
ids.append(record["id"])
to_write = new_records

with singer.metrics.record_counter(endpoint=table) as ctr:
singer.write_records(table, to_write)
Expand Down