Release v6.0.1

This release contains a small fix to correct an incompatibility between the 6.0.0 pinecone release and pinecone-plugin-assistant. While working toward improving type coverage of the sdk, some attributes of the internal Configuraiton class were erroneously removed even though they are still needed by the plugin to load correctly.

The 6.0.0 pinecone SDK should now work with all versions of pinecone-plugin-assistant except for 1.1.1 which errors when used with 6.0.0.

Thanks @avi1mizrahi for contributing the fix.

Release v6.0.0

What's new in this release?

Indexes with Integrated Inference

This release adds a new create_index_for_model method as well as upsert_records, and search methods. Together these methods provide a way for you to easily store your data and let us manage the process of creating embeddings. To learn about available models, see the Model Gallery.

Note: If you were previously using the preview versions of this functionality via the pinecone-plugin-records package, you will need to uninstall that package in order to use the v6 pinecone release.

from pinecone import (
    Pinecone,
    CloudProvider,
    AwsRegion,
    EmbedModel,
)

# 1. Instantiate the Pinecone client
pc = Pinecone(api_key="<<PINECONE_API_KEY>>")

# 2. Create an index configured for use with a particular model
index_config = pc.create_index_for_model(
    name="my-model-index",
    cloud=CloudProvider.AWS,
    region=AwsRegion.US_EAST_1,
    embed=IndexEmbed(
        model=EmbedModel.Multilingual_E5_Large,
        field_map={"text": "my_text_field"}
    )
)

# 3. Instantiate an Index client
idx = pc.Index(host=index_config.host)

# 4. Upsert records
idx.upsert_records(
    namespace="my-namespace",
    records=[
        {
            "_id": "test1",
            "my_text_field": "Apple is a popular fruit known for its sweetness and crisp texture.",
        },
        {
            "_id": "test2",
            "my_text_field": "The tech company Apple is known for its innovative products like the iPhone.",
        },
        {
            "_id": "test3",
            "my_text_field": "Many people enjoy eating apples as a healthy snack.",
        },
        {
            "_id": "test4",
            "my_text_field": "Apple Inc. has revolutionized the tech industry with its sleek designs and user-friendly interfaces.",
        },
        {
            "_id": "test5",
            "my_text_field": "An apple a day keeps the doctor away, as the saying goes.",
        },
        {
            "_id": "test6",
            "my_text_field": "Apple Computer Company was founded on April 1, 1976, by Steve Jobs, Steve Wozniak, and Ronald Wayne as a partnership.",
        },
    ],
)

# 5. Search for similar records
from pinecone import SearchQuery, SearchRerank, RerankModel

response = index.search_records(
    namespace="my-namespace",
    query=SearchQuery(
        inputs={
            "text": "Apple corporation",
        },
        top_k=3
    ),
    rerank=SearchRerank(
        model=RerankModel.Bge_Reranker_V2_M3,
        rank_fields=["my_text_field"],
        top_n=3,
    ),
)

Call the Inference API

You can now interact with Pinecone's Inference API without the need to install any extra plugins.

Note: If you were previously using the preview versions of this functionality via the pinecone-plugin-inference package, you will need to uninstall that package.

from pinecone import Pinecone

pc = Pinecone(api_key="<<PINECONE_API_KEY>>")

inputs = ["Who created the first computer?"]
outputs = pc.inference.embed(
    model="multilingual-e5-large", 
    inputs=inputs, parameters={"input_type": "passage", "truncate": "END"}
)
print(outputs)
#  EmbeddingsList(
#      model='multilingual-e5-large',
#      data=[
#          {'values': [0.1, ...., 0.2]},
#        ],
#      usage={'total_tokens': 6}
#  )

New client variants with support for asyncio

The v6 Python SDK introduces a new client variants, PineconeAsyncio and IndexAsyncio, which provide async methods for use with asyncio. This should unblock those who wish to use Pinecone with modern async web frameworks such as FastAPI, Quart, Sanic, etc. Those trying to onboard to Pinecone and upsert large amounts of data should significantly benefit from the efficiency of running many upserts in parallel.

To use these, you will need to install pinecone[asyncio] which pulls in an extra depdency on aiohttp. See notes on installation.

You can expect more documentation and information on how to use these asyncio clients to follow soon.

import asyncio

from pinecone import (
    PineconeAsyncio,
    IndexEmbed,
    CloudProvider,
    AwsRegion,
    EmbedModel
)

async def main():
    async with PineconeAsyncio() as pc:
        if not await pc.has_index(index_name):
            desc = await pc.create_index_for_model(
                name="book-search",
                cloud=CloudProvider.AWS,
                region=AwsRegion.US_EAST_1,
                embed=IndexEmbed(
                    model=EmbedModel.Multilingual_E5_Large,
                    metric="cosine",
                    field_map={
                        "text": "description",
                    },
                )
            )

asyncio.run(main())

Interactions with a deployed index are done via IndexAsyncio class, which can be instantiated using helper methods on either Pinecone or PineconeAsyncio:

import asyncio
from pinecone import Pinecone

async def main():
    pc = Pinecone(api_key='<<PINECONE_API_KEY>>')    
    async with pc.IndexAsyncio(host="book-search-dojoi3u.svc.aped-4627-b74a.pinecone.io") as idx:
        await idx.upsert_records(
            namespace="books-records",
            records=[
                {
                    "id": "1",
                    "title": "The Great Gatsby",
                    "author": "F. Scott Fitzgerald",
                    "description": "The story of the mysteriously wealthy Jay Gatsby and his love for the beautiful Daisy Buchanan.",
                    "year": 1925,
                },
                {
                    "id": "2",
                    "title": "To Kill a Mockingbird",
                    "author": "Harper Lee",
                    "description": "A young girl comes of age in the segregated American South and witnesses her father's courageous defense of an innocent black man.",
                    "year": 1960,
                },
                {
                    "id": "3",
                    "title": "1984",
                    "author": "George Orwell",
                    "description": "In a dystopian future, a totalitarian regime exercises absolute control through pervasive surveillance and propaganda.",
                    "year": 1949,
                },
            ]
        )


asyncio.run(main())

Organize your indexes with tags

Tags are key-value pairs you can attach to indexes to better understand, organize, and identify your resources. Tags are flexible and can be tailored to your needs, but some common use cases for them might be to label an index with the relevant deployment environment, application, team, or owner.

Tags can be set during index creation by passing an optional dictionary with the tags keyword argument to the create_index and create_index_for_model methods. Here's an example demonstrating how tags can be passed to create_index.

from pinecone import (
    Pinecone,
    ServerlessSpec,
    CloudProvider,
    GcpRegion,
    Metric
)

pc = Pinecone(api_key='<<PINECONE_API_KEY>>')

pc.create_index(
    name='my-index',
    dimension=1536,
    metric=Metric.COSINE,
    spec=ServerlessSpec(
        cloud=CloudProvider.GCP,
        region=GcpRegion.US_CENTRAL1
    ),
    tags={
        "environment": "testing",
        "owner": "jsmith",
    }
)

See this page for more documentation about how to add, modify, or remove tags.

Sparse indexes early access support

Sparse indexes are currently in early access. This release will allow those with early access to create sparse indexes and view those configurations with the describe_index and list_indexes methods.

These are created using the same create_index method as other index types but with different configuration options. For sparse indexes, you must omit dimension while passingmetric="dotproduct" and vector_type="sparse".

from pinecone import (
    Pinecone,
    ServerlessSpec,
    CloudProvider,
    AwsRegion,
    Metric,
    VectorType
)

pc = Pinecone()
pc.create_index(
    name='sparse-index',
    metric=Metric.DOTPRODUCT,
    spec=ServerlessSpec(
        cloud=CloudProvider.AWS,
        region=AwsRegion.US_WEST_2
    ),
    vector_type=VectorType.SPARSE
)

# Check the description to get the host url
desc = pc.describe_index(name='sparse-index')

# Instantiate the index client
sparse_index = pc.Index(host=desc.host)

Upserting and querying a sparse index is very similar to before, except now the values field of a Vector (used when working with dense values) may be unset.

import random
from pinecone import Vector, SparseValues

def unique_random_integers(n, range_start, range_end):
    if n > (range_end - range_start + 1):
        raise ValueError("Range too small for the requested number of unique integers")
    return random.sample(range(range_start, range_end + 1), n)

# Generate some random sparse vectors
sparse_index.upsert(
    vectors=[
        Vector(
            id=str(i),
            sparse_values=SparseValues(
                indices=unique_random_integers(10, 0, 10000),
                values=[random.random() for j in range(10)]
            )
        ) for i in range(10000)
    ],...

Release v5.4.2

This release contains a small adjustment to the query_namespaces method added in the 5.4.0. The initial implementation had a bug that meant it could not properly merge small result sets across multiple namespaces. This release adds a required keyword argument, metric to the query_namespaces method, which should enable the SDK to merge results no matter how many results are returned.

from pinecone import Pinecone

pc = Pinecone(api_key='YOUR_API_KEY')
index = pc.Index(host='your-index-host')

query_results = index.query_namespaces(
    vector=[0.1, 0.2, ...],  # The query vector, dimension should match your index
    namespaces=['ns1', 'ns2', 'ns3'],
    metric="cosine", # This is the new required keyword argument
    include_values=False,
    include_metadata=True,
    filter={},
    top_k=100,
)

What's Changed

• [Bug] query_namespaces can handle single result by @jhamon in #421

Full Changelog: v5.4.1...v5.4.2

Release v5.4.1

What's Changed

• [Chore] Allow support for pinecone-plugin-inference >=2.0.0, <4.0.0 by @austin-denoble in #419

Release v5.4.0

Query namespaces

In this release we have added a utility method to run a query across multiple namespaces, then merge the result sets into a single ranked result set with the top_k most relevant results. The query_namespaces method accepts most of the same arguments as query with the addition of a required namespaces param.

Since query_namespaces executes multiple queries in parallel, in order to get good performance it is important to set values for the pool_threads and connection_pool_maxsize properties on the index client. The pool_threads setting is the number of threads available to execute requests while connection_pool_maxsize is the number of cached http connections that will be held. Since these tasks are not computationally heavy and are mainly i/o bound, it should be okay to have a high ratio of threads to cpus.

The combined results include the sum of all read unit usage used to perform the underlying queries for each namespace.

from pinecone import Pinecone

pc = Pinecone(api_key="key")
index = pc.Index(
  name="index-name",
  pool_threads=50,             # <-- make sure to set these
  connection_pool_maxsize=50,  # <-- make sure to set these
)

query_vec = [ 0.1, ...] # an embedding vector with same dimension as the index
combined_results = index.query_namespaces(
    vector=query_vec,
    namespaces=['ns1', 'ns2', 'ns3', 'ns4'],
    top_k=10,
    include_values=False,
    include_metadata=True,
    filter={"genre": { "$eq": "comedy" }},
    show_progress=False,
)

for scored_vec in combined_results.matches:
    print(scored_vec)
print(combined_results.usage)

A version of query_namespaces is also available over grpc. For grpc, there is no need to set the connection_pool_maxsize because grpc makes efficient use of open connections by default.

from pinecone.grpc import PineconeGRPC

pc = PineconeGRPC(api_key="key")
index = pc.Index(
  name="index-name",
  pool_threads=50, # <-- make sure to set this
)

query_vec = [ 0.1, ...] # an embedding vector with same dimension as the index
combined_results = index.query_namespaces(
    vector=query_vec,
    namespaces=['ns1', 'ns2', 'ns3', 'ns4'],
    top_k=10,
    include_values=False,
    include_metadata=True,
    filter={"genre": { "$eq": "comedy" }},
    show_progress=False,
)

for scored_vec in combined_results.matches:
    print(scored_vec)
print(combined_results.usage)

Changelog

Additions

• [feat] PineconeGrpcFuture implements concurrent.futures.Future by @jhamon in #410
• Update to pinecone-plugin-inference=2.0.0 by @ssmith-pc in #397
• Detect plugins for Index and IndexGRPC classes by @jhamon in #402
• Add query_namespaces by @jhamon in #409
• Expose connection_pool_maxsize on Index and add docstrings by @jhamon in #415
• Implement query_namespaces over grpc by @jhamon in #416
• query_namespaces performance improvements by @jhamon in #417

Chores / Fixes

• [Refactor] Extract GrpcChannelFactory from GRPCIndexBase by @jhamon in #394
• [Refactor] Extract GrpcRunner from GRPCIndexBase class by @jhamon in #395
• [Chore] Replace black with ruff linter / formatter by @jhamon in #392
• [Fix] Update build-oas script for building exceptions template changes by @ssmith-pc in #396
• [Chore] Put date into test index and collection names by @jhamon in #399
• [Chore] Automatically cleanup old resources each night by @jhamon in #400
• [Chore] Improve test flakes by @jhamon in #404

Full Changelog: v5.3.1...v5.4.0.dev5

Release v5.3.1

What's Changed

• [Fix] Add missing python-dateutil dependency by @jhamon in #391

Release v5.3.0

Public Preview: Imports

To learn more about working with imports and details about expected data formats, please see these documentation guides:

• Understanding Imports
• Import Data
• Manage Storage Integrations

This release adds methods for interacting with several new endpoints in Public Preview from the Python SDK. Before you can use these, you will need to follow the above docs to prepare your data and configure any storage integrations.

import os
import random
from pinecone import Pinecone, ServerlessSpec, ImportErrorMode

# 0. Instantiate your client instance
pc = Pinecone(api_key=os.environ['PINECONE_API_KEY'])

# 1. You must have an index whose dimension matches the size of your data
#    You may already have such an index, but for this demo we will create one.
index_name = f"import-{random.randint(0, 10000)}"

if not pc.has_index(index_name):
  pc.create_index(
      name=index_name,
      dimension=10,
      metric="cosine",
      spec=ServerlessSpec(cloud="aws", region="eu-west-1")
  )

# 2. Get a reference to the index client
index = pc.Index(name=index_name)

# 3. Start the import operation, passing a uri that describes the path to your
#    AWS S3 bucket. Each subfolder within this path will correspond to a namespace
#    where imported data will be stored.
root = 's3://dev-bulk-import-datasets-pub/10-records-dim-10/'
op = index.start_import(
    uri=root,
    error_mode=ImportErrorMode.CONTINUE, # or ABORT
    # integration_id='' # Add this if you want to use a storage integration
  )

# 4. Check the operation status
index.describe_import(id=op.id)

# 5. Cancel an import operation
index.cancel_import(id=op.id)

# 6. List all recent operations using a generator that handles pagination on your behalf
for i in index.list_imports():
  print(f"id: {i.id} status: {i.status}")

# ...or turn the generator into a simple list, fetching all results at once
operations = list(index.list_imports())
print(operations)

Release v5.2.0

Public Preview: Rerank

This release adds a method for interacting with our Rerank endpoint, now in Public Preview. Rerank is used to order results by relevance to a query.

Currently rerank supports the bge-reranker-v2-m3 model. See the rerank guide for more information on using this feature.

from pinecone import Pinecone

pc = Pinecone(api_key="your api key")

query = "Tell me about Apple's products"

results = pc.inference.rerank(
    model="bge-reranker-v2-m3",
    query=query,
    documents=[
      "Apple is a popular fruit known for its sweetness and crisp texture.",
      "Apple is known for its innovative products like the iPhone.",
      "Many people enjoy eating apples as a healthy snack.",
      "Apple Inc. has revolutionized the tech industry with its sleek designs and user-friendly interfaces.",
      "An apple a day keeps the doctor away, as the saying goes.",
    ],
    top_n=3,
    return_documents=True,
)

print(query)
for r in results.data:
  print(r.score, r.document.text)

Gives output along these lines

Tell me about Apple's products
0.8401279 Apple is known for its innovative products like the iPhone.
0.23318209 Apple Inc. has revolutionized the tech industry with its sleek designs and user-friendly interfaces.
0.17384852 Apple is a popular fruit known for its sweetness and crisp texture.

Release v5.1.0

Package renamed from pinecone-client to pinecone

In this release, we have renamed the package from pinecone-client to pinecone. From now on you should install it using the pinecone name.

There is a plan to continue publishing code under the pinecone-client package as well so that anyone using the old name will still find out about available upgrades via their dependency management tool of choice, but we haven't automated that as part of our release process yet so there will be a slight delay in new work being released under that name.

New has_index() helper and improved output

We've added a small helper function to simplify a common need in notebooks and examples, which is checking if an index exists.

from pinecone import Pinecone, ServerlessSpec

pc = Pinecone(api_key='YOUR_API_KEY')

index_name = "movie-recommendations"

if not pc.has_index(index_name):
    pc.create_index(
        name=index_name,
        dimension=384,
        metric="cosine",
        spec=ServerlessSpec(cloud="aws", region="us-west-2")
    )

index = pc.Index(name=index_name)

# Now upsert vectors, run queries, etc

If you are frequently working in notebooks, you will also benefit from a nicer presentation of control plane responses.

>>> pc.describe_index(name="test-embed2")
{
    "name": "test-embed2",
    "dimension": 10,
    "metric": "cosine",
    "host": "test-embed2-dojoi3u.svc.apw5-4e34-81fa.pinecone.io",
    "spec": {
        "serverless": {
            "cloud": "aws",
            "region": "us-west-2"
        }
    },
    "status": {
        "ready": true,
        "state": "Ready"
    },
    "deletion_protection": "disabled"
}

What's Changed

• [Docs] Fix dataframe column name in doc strings by @jseldess in #381
• [Docs] Change "client" to "SDK" in README by @jseldess in #382
• [Chore] Adding new issue templates by @anawishnoff in #380
• [Chore] Reduce dimension in testing to simplify output by @jhamon in #384
• [Chore] Rename package from pinecone-client to pinecone by @jhamon in #383
• [Feature] Add has_index() by @rohanshah18 in #385
• [Feature] Improve output from list/describe actions on indexes and collections by @jhamon in #387

New Contributors

• @jseldess made their first contribution in #381
• @anawishnoff made their first contribution in #380
• @rohanshah18 made their first contribution in #385

Full Changelog: v5.0.1...v5.1.0

Release v5.0.1

What's Changed

• [CI] Publish doc updates after each release by @jhamon in #373
• [Fix] Fetch when vector id string contains spaces by @jhamon in #372
• [Fix] Adjusting inference plugin dependency to resolve circular dependency by @jhamon @ssmith-pc in #379 #377

Full Changelog: v5.0.0...v5.0.1