Skip to content
This repository has been archived by the owner on Apr 11, 2024. It is now read-only.

Handling conflicts and errors for dataset transfers #27

Open
9 tasks
sunank200 opened this issue Mar 27, 2023 · 0 comments
Open
9 tasks

Handling conflicts and errors for dataset transfers #27

sunank200 opened this issue Mar 27, 2023 · 0 comments

Comments

@sunank200
Copy link
Collaborator

Please describe the feature you'd like to see
There are multiple scenarios to handle dataset transfers:

  1. Lets take a scenario where we are transferring 100 rows from snowflake to bigquery and error occurs after transferring the 99th row. What's the correct way to handle this?
    • By default should this be a TRANSACTION and roll back all the changes done?
    • Or should it throw the error for all the rows that error out? What's the right way to error out for a larger number of rows that has the error

This should be handled before 1.0 release. To start with we can replicate something similar to Fivetran does.

Acceptance Criteria

  • All checks and tests in the CI should pass
  • Unit tests (90% code coverage or more)
  • Integration tests (if the feature relates to a new database or external service)
  • Example DAG
  • Docstrings in reStructuredText for each of methods, classes, functions and module-level attributes (including Example DAG on how it should be used)
  • Exception handling in case of errors
  • Logging (are we exposing useful information to the user? e.g. source and destination)
  • Improve the documentation (README, Sphinx, and any other relevant)
  • How to use Guide for the feature (example)
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants