How Do I Create & Upload Data To A Vector Index Of A Different Dimension Than 1536

[sfdrada]

GitHub Issue: Support for Increased Vector Dimensions in Pinecone Canopy

Description

The default dimensions for embedding models in Pinecone Canopy are set to 1536. However, the new OpenAI embedding model text-embedding-3-large supports 3072 dimensions. Attempts to adjust the dimensions in the default.yaml configuration file have not led to the creation of a new Pinecone database with the updated dimensions.

When setting up the Pinecone Canopy to run an index with 3072 dimensions by first creating it manually then redirecting the canopy command to the name of the vector index, and attempting to upsert to it through Canopy, an error is encountered stating that the vector index does not match the 1536 dimension specification.

Below is a snippet of the current default.yaml settings:

record_encoder:
  type: OpenAIRecordEncoder
  params:
    model_name: text-embedding-3-large
    batch_size: 400
    dimension: 3072
    Dimension: 3072
    dimensions: 3072
    Dimensions: 3072

create_index_params:
  metric: cosine
  spec:
    serverless:
      cloud: aws
      region: us-west-2
      dimension: 3072
      Dimension: 3072
      dimensions: 3072
      Dimensions: 3072
    # Additional configuration...

Request

All of the above 'dimensions' variables were manually added and were not a part of the original yaml structure.

I request assistance in modifying the Pinecone Canopy configuration to support the 3072 dimensions required by the text-embedding-3-large model. Specifically, guidance on how to properly configure the default.yaml file or any other necessary steps to achieve compatibility with the larger dimension size is needed.

Any assistance or insights from the community or the Pinecone development team would be greatly appreciated.