Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Update the
target-postgres
dependency to get the latest master branch commit, which is many commits ahead of the 0.2.4 package version. This provides several performance optimizations that have been added, such as:Additionally, bypass table insertion work when the record count is zero. The code still respects the
persist_empty_tables
setting to manage the table schema itself, but will not go through the expensive process to perform a zero-record insertion. That process takes ~6s per table and is effectively a no-op. For table-heavy taps, this saves a lot of wasted time.Testing
Tested locally with a few of my own taps. A tap with ~25 tables and no new data was able to run in ~40 instead of ~1m45s, mostly due to the code which skips table data insertion when the data batch has zero records.