Skip to content

Latest commit

 

History

History
99 lines (68 loc) · 4.27 KB

readme.md

File metadata and controls

99 lines (68 loc) · 4.27 KB

S3 Sync OpenAI Vector Store

Sync the contents of an S3 bucket with an OpenAI vector store

made with hearth by Eyevinn Slack

Sync the contents of an S3 bucket with an OpenAI vector store

Badge OSC

Requirements

  • OpenAI API key
  • AWS S3 CLI

Installation / Usage

Options can be provided either as command line options or environment variables.

% npx @eyevinn/s3-sync-vectorstore -h
Sync the files on an S3 bucket with an OpenAI vector store

Arguments:
  source                                   Source S3 URL
  vector-store-id                          OpenAI vector store ID

Options:
  --openai-api-key                         OpenAI API key (OPENAI_API_KEY)
  --purpose                                Purpose of the files (OPENAI_PURPOSE)
  --region <region>                        AWS region (AWS_REGION)
  --endpoint <endpoint>                    S3 endpoint (S3_ENDPOINT)
  --access-key-id <access-key-id>          AWS access key ID (AWS_ACCESS_KEY_ID)
  --secret-access-key <secret-access-key>  AWS secret access key (AWS_SECRET_ACCESS_KEY)
  --staging-dir <staging-dir>              Staging directory (STAGING_DIR) (default: "/tmp/data")
  --dry-run                                Dry run
  -h, --help                               display help for command

% export OPENAI_API_KEY=<openai-api-key>
% npm start s3://source/files/ vs_3434533213

You can also sync multiple S3 buckets with one vector store by comma separating the source URLs

% npm start s3://source/files/,s3://source2/files/ vs_3434533213

Running in Eyevinn Open Source Cloud

Available as an open web service in Eyevinn Open Source Cloud if you want to run a job in the cloud without having to setup your own infrastructure for it.

% export OSC_ACCESS_TOKEN=<your-osc-pat>
% npx -y @osaas/cli@latest create eyevinn-s3-sync-vectorstore ghsync \
    -o cmdLineArgs="s3://source/files vs_3434533213" \
    -o OpenaiApiKey="{{secrets.openaikey}}" \
    -o AwsRegion="eu-north-1" \
    -o AwsAccessKeyId="{{secrets.accesskeyid}}" \
    -o AwsSecretAccessKey="{{secrets.secretaccesskey}}"

Contributing

See CONTRIBUTING

License

This project is licensed under the MIT License, see LICENSE.

Support

Join our community on Slack where you can post any questions regarding any of our open source projects. Eyevinn's consulting business can also offer you:

  • Further development of this component
  • Customization and integration of this component into your platform
  • Support and maintenance agreement

Contact [email protected] if you are interested.

About Eyevinn Technology

Eyevinn Technology is an independent consultant firm specialized in video and streaming. Independent in a way that we are not commercially tied to any platform or technology vendor. As our way to innovate and push the industry forward we develop proof-of-concepts and tools. The things we learn and the code we write we share with the industry in blogs and by open sourcing the code we have written.

Want to know more about Eyevinn and how it is to work here. Contact us at [email protected]!