batch upsert by chunks in cli #132

acatav · 2023-11-01T11:21:02Z

Problem

batch upsert should be determined by the number of chunks and not number of documents for large upsert

For now we will batch by number of chunks only in the CLI. Later we will refactor the whole upsert pipeline

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update
Infrastructure change (CI configs, etc)
Non-code change (docs, etc)
None of the above: (explain here)

Manual tests with short documents dataset, and with long documents dataset

igiloh-pinecone

LGTM. See a few comments

src/canopy_cli/cli.py

batch upsert by chunks in cli

b1bb6c2

igiloh-pinecone approved these changes Nov 1, 2023

View reviewed changes

src/canopy_cli/cli.py Outdated Show resolved Hide resolved

src/canopy_cli/cli.py Outdated Show resolved Hide resolved

acatav added 2 commits November 1, 2023 15:14

remove batch size from upsert params in cli

99c3560

change type

9396252

acatav enabled auto-merge November 1, 2023 13:26

acatav added this pull request to the merge queue Nov 1, 2023

Merged via the queue into dev with commit 10c0e7d Nov 1, 2023
10 checks passed

acatav deleted the cli-batch-upsert-by-chunks branch November 1, 2023 13:45