Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support bulk imports in import_without_validations_or_callbacks #84

Open
colinmroberts opened this issue Aug 10, 2020 · 0 comments
Open
Labels
enhancement New feature or request

Comments

@colinmroberts
Copy link
Collaborator

When dealing with bulk imports for DB Backend records, we should add better support for refreshing records from the DB. Specifically, in import_without_validations_or_callbacks in kafka_source.rb, we currently do a single DB query for each record that is being imported. When there is a bulk import, especially with thousands of records, this is extremely slow.

If we could bulk-query the list of IDs and match them up with the records being imported, this would make this import much more efficient

@colinmroberts colinmroberts added the enhancement New feature or request label Aug 10, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant