Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Persist rejected data table #293

Open
jeremyprime opened this issue Dec 8, 2021 · 0 comments
Open

Persist rejected data table #293

jeremyprime opened this issue Dec 8, 2021 · 0 comments
Labels

Comments

@jeremyprime
Copy link
Collaborator

jeremyprime commented Dec 8, 2021

The rejected data table is a temporary table, and due to our implementation it is never persisted to Vertica. See the documentation regarding the rejected data table and the use of COPY ... NO COMMIT here.

This ticket is to persist the rejected data table, but without affecting our current logic around committing or not committing data based on the error threshold. The rejected data should be saved whenever there is at least one rejected row, but this should not affect whether we commit or do not commit the actual data.

We should also consider adding connector options to enable this functionality and specify the table name for the rejected data.
For example, we could add a persist_rejected_data option (defaults to true) and a rejected_data_table option (defaults to <table>_<session_id>_REJECTS).

Note that we added some logging of rejected table data under #275.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants