Skip to content

Latest commit

 

History

History
37 lines (26 loc) · 3.23 KB

setup.md

File metadata and controls

37 lines (26 loc) · 3.23 KB
Raw

Pinecone

Pinecone is a managed vector database built for speed, scale, and shipping to production sooner. To use Pinecone as your vector database provider, first get an API key by signing up for an account. You can access your API key from the "API Keys" section in the sidebar of your dashboard. Pinecone also supports hybrid search and at the time of writing is the only datastore to support SPLADE sparse vectors natively.

A full Jupyter notebook walkthrough for the Pinecone flavor of the retrieval plugin can be found here. There is also a video walkthrough here.

The app will create a Pinecone index for you automatically when you run it for the first time. Just pick a name for your index and set it as an environment variable.

Environment Variables:

Name Required Description
DATASTORE Yes Datastore name, set this to pinecone
BEARER_TOKEN Yes Your secret token for authenticating requests to the API
OPENAI_API_KEY Yes Your OpenAI API key for generating embeddings with one of the OpenAI embeddings models
PINECONE_API_KEY Yes Your Pinecone API key, found in the Pinecone console
PINECONE_ENVIRONMENT Yes Your Pinecone environment, found in the Pinecone console, e.g. us-west1-gcp, us-east-1-aws, etc.
PINECONE_INDEX Yes Your chosen Pinecone index name. Note: Index name must consist of lower case alphanumeric characters or '-'

If you want to create your own index with custom configurations, you can do so using the Pinecone SDK, API, or web interface (see docs). Make sure to use a dimensionality of 256 (or another dimension) for the embeddings and avoid indexing on the text field in the metadata, as this will reduce the performance significantly.

# Creating index with Pinecone SDK - use only if you wish to create the index manually.

import os, pinecone

pinecone.init(api_key=os.environ['PINECONE_API_KEY'],
              environment=os.environ['PINECONE_ENVIRONMENT'])

EMBEDDING_DIMENSION = int(os.environ.get("EMBEDDING_DIMENSION", 256))

pinecone.create_index(name=os.environ['PINECONE_INDEX'],
                      dimension=EMBEDDING_DIMENSION,
                      metric='cosine',
                      metadata_config={
                          "indexed": ['source', 'source_id', 'url', 'created_at', 'author', 'document_id']})