Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add milvus bulk #190

Open
wants to merge 37 commits into
base: main
Choose a base branch
from
Open

Add milvus bulk #190

wants to merge 37 commits into from

Commits on Oct 15, 2024

  1. add in gpu milvus

    @jperez999
    jperez999 committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    3bdcb1c View commit details
    Browse the repository at this point in the history

Commits on Oct 21, 2024

  1. fix image storage extra params pipe

    @jperez999
    jperez999 committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    1f8a29d View commit details
    Browse the repository at this point in the history

  2. add changes for store embedding task

    @jperez999
    jperez999 committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    743220d View commit details
    Browse the repository at this point in the history

  3. add embedding schema and ingestion logic

    @jperez999
    jperez999 committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    14fb6de View commit details
    Browse the repository at this point in the history

  4. add embedding upload logic for store embedding task

    @jperez999
    jperez999 committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    6498a1a View commit details
    Browse the repository at this point in the history

  5. add store embedding task to pipeline and for commandline

    @jperez999
    jperez999 committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    aeb7a46 View commit details
    Browse the repository at this point in the history

  6. add s3fs for direct upload to bucket for pandas

    @jperez999
    jperez999 committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    3f454a8 View commit details
    Browse the repository at this point in the history

  7. update tests for store and add tests for store embed

    @jperez999
    jperez999 committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    af21a90 View commit details
    Browse the repository at this point in the history

  8. add changes to allow for safe usage in cli

    @jperez999
    jperez999 committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    2b8e480 View commit details
    Browse the repository at this point in the history

  9. add changes to client example notebook

    @jperez999
    jperez999 committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    73748a1 View commit details
    Browse the repository at this point in the history

  10. add bucket path param for extra params

    @jperez999
    jperez999 committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    ea52f86 View commit details
    Browse the repository at this point in the history

  11. remove outputs from notebook

    @jperez999
    jperez999 committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    91e2339 View commit details
    Browse the repository at this point in the history

  12. merge in main

    @jperez999
    jperez999 committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    3cba7de View commit details
    Browse the repository at this point in the history

Commits on Oct 23, 2024

  1. remove logger messages

    @jperez999
    jperez999 committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    f33fc99 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into minio-embedding-upload

    @jperez999
    jperez999 authored Oct 23, 2024
    Configuration menu
    Copy the full SHA
    9c03d7e View commit details
    Browse the repository at this point in the history

Commits on Oct 24, 2024

  1. removed unnecessary logging and commented code

    @jperez999
    jperez999 committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    f34e5cf View commit details
    Browse the repository at this point in the history

  2. Merge branch 'minio-embedding-upload' of https://github.com/jperez999… 

    …/nv-ingest into minio-embedding-upload
    @jperez999
    jperez999 committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    95d0e02 View commit details
    Browse the repository at this point in the history

  3. add bulk ingest support for minio loaded records

    @jperez999
    jperez999 committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    3c36548 View commit details
    Browse the repository at this point in the history

  4. merge in main

    @jperez999
    jperez999 committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    d1d539a View commit details
    Browse the repository at this point in the history

  5. change args to params for standardization

    @jperez999
    jperez999 committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    ea0f13d View commit details
    Browse the repository at this point in the history

  6. Update client/src/nv_ingest_client/cli/util/click.py 

    Co-authored-by: Devin Robison <[email protected]>
    @jperez999 @drobison00
    jperez999 and drobison00 authored Oct 24, 2024
    Configuration menu
    Copy the full SHA
    c83a0f1 View commit details
    Browse the repository at this point in the history

  7. Update client/src/nv_ingest_client/cli/util/click.py 

    Co-authored-by: Devin Robison <[email protected]>
    @jperez999 @drobison00
    jperez999 and drobison00 authored Oct 24, 2024
    Configuration menu
    Copy the full SHA
    30ec521 View commit details
    Browse the repository at this point in the history

  8. Update client/src/nv_ingest_client/primitives/tasks/__init__.py 

    Co-authored-by: Devin Robison <[email protected]>
    @jperez999 @drobison00
    jperez999 and drobison00 authored Oct 24, 2024
    Configuration menu
    Copy the full SHA
    fe6954e View commit details
    Browse the repository at this point in the history

  9. Update client/src/nv_ingest_client/primitives/tasks/task_factory.py 

    Co-authored-by: Devin Robison <[email protected]>
    @jperez999 @drobison00
    jperez999 and drobison00 authored Oct 24, 2024
    Configuration menu
    Copy the full SHA
    f56c9b4 View commit details
    Browse the repository at this point in the history

Commits on Oct 29, 2024

  1. add progress engines to vdb task sink

    @jperez999
    jperez999 committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    80257e6 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'add-milvus-bulk' of https://github.com/jperez999/nv-ingest 

     into add-milvus-bulk
    @jperez999
    jperez999 committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    6d05bfc View commit details
    Browse the repository at this point in the history

  3. merge main in

    @jperez999
    jperez999 committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    1e5cd8d View commit details
    Browse the repository at this point in the history

Commits on Oct 30, 2024

  1. made the embedding storage stage multiprocess

    @jperez999
    jperez999 committed Oct 30, 2024
    Configuration menu
    Copy the full SHA
    35a734e View commit details
    Browse the repository at this point in the history

  2. add in updated store_embed task schema

    @jperez999
    jperez999 committed Oct 30, 2024
    Configuration menu
    Copy the full SHA
    d02f329 View commit details
    Browse the repository at this point in the history

Commits on Nov 18, 2024

  1. Merge branch 'main' into add-milvus-bulk

    @jperez999
    jperez999 authored Nov 18, 2024
    Configuration menu
    Copy the full SHA
    8b51108 View commit details
    Browse the repository at this point in the history

  2. fix tests and cli interface

    @jperez999
    jperez999 committed Nov 18, 2024
    Configuration menu
    Copy the full SHA
    a1cd956 View commit details
    Browse the repository at this point in the history

  3. replace params for extra_params in test for job ingest schema

    @jperez999
    jperez999 committed Nov 18, 2024
    Configuration menu
    Copy the full SHA
    f020424 View commit details
    Browse the repository at this point in the history

  4. revert to params field to comply with current usage

    @jperez999
    jperez999 committed Nov 18, 2024
    Configuration menu
    Copy the full SHA
    2fb6062 View commit details
    Browse the repository at this point in the history

  5. full push of all schema changes for ingest about params

    @jperez999
    jperez999 committed Nov 18, 2024
    Configuration menu
    Copy the full SHA
    074476e View commit details
    Browse the repository at this point in the history

  6. fix tests to reflect params revision

    @jperez999
    jperez999 committed Nov 18, 2024
    Configuration menu
    Copy the full SHA
    69da980 View commit details
    Browse the repository at this point in the history

Commits on Nov 21, 2024

  1. revamped parquet table embedding and vdb upload

    @jperez999
    jperez999 committed Nov 21, 2024
    Configuration menu
    Copy the full SHA
    8d7ca17 View commit details
    Browse the repository at this point in the history

  2. use params to load in collection name to embedding storage task

    @jperez999
    jperez999 committed Nov 21, 2024
    Configuration menu
    Copy the full SHA
    8ddebc1 View commit details
    Browse the repository at this point in the history