Release v4.1.1

Fixes

• Allow colon inside source tags by @jhamon in #351
• [grpc] Allow retries of up to MAX_MSG_SIZE by @daverigby in #347

Dependabot security updates

• Bump requests (dev-only dependency) from 2.31.0 to 2.32.3 by @jhamon in #352

Chores

• Remove unused constants by @jhamon in #350
• Wire up plugin interface by @jhamon in #353

Full Changelog: v4.1.0...v4.1.1

Release v4.1.0

Features

• Support proxy_url and ssl_ca_certs options for gRPC by @daverigby in #341
• Add better error messages for mistaken from_texts and from_documents invocations by @jhamon in #342

Dependabot security fixes

• Bump jinja2 from 3.1.3 to 3.1.4 by @jhamon in #343
• Bump idna from 3.4 to 3.7 by @jhamon in #345
• Bump tqdm from 4.66.1 to 4.66.3 by @jhamon in #344

Full Changelog: v4.0.0...v4.1.0

Release v4.0.0

3x boost to upsert throughput via grpc

In this release, we are upgrading theprotobuf dependency in our optional grpc extras from 3.20.3 to 4.25.3.

This is a breaking change for users of the optional GRPC addon (installed with pinecone-client[grpc]). Detailed performance profiling by @daverigby showed this dependency upgrade unlocked a 3x improvement to vector upsert performance in the Python SDK, bringing it much closer to the performance observed in our grpc-based Java SDK.

As a reminder, to use use the optional grpc extras for improved performance, you need to install pinecone-client[grpc] and make a small modification to how you import and use the client. See the instructions for doing this here.

What's Changed

• Improve grpc upsert throughput by 3x by @daverigby in #334

Release v3.2.2

Bug fixes

This small release includes a fix for a minor issue introduced last week in the v3.2.0 release that resulted in a DeprecationWarning being incorrectly shown to users who are not passing in the deprecated openapi_config property. This warning notice can safely be ignored by anyone who isn't prepared to upgrade.

Thanks to riku in Pinecone's Community Forum for reporting this issue.

• [Fix] openapi_config deprecation warning incorrectly shown by @jhamon in #327

Chores

• [Chore] Run grpc unit tests in CI, expand testing of VectorFactory by @jhamon in #326

Release v3.2.1

What's Changed

• Allow clients to tag requests with a source_tag by @ssmith-pc in #324

Source tag

The SDK now optionally allows setting a source tag when constructing a Pinecone client. The source tag allows requests to be associated with the source tag provided.

from pinecone import Pinecone

pc = Pinecone(api_key='your-key', source_tag='foo')

# requests initiated from pc connection are associated with source tag "foo"

or

from pinecone.grpc import PineconeGRPC

pc = PineconeGRPC(api_key='your-key', source_tag='foo')

New Contributors

• @ssmith-pc made their first contribution in #324

Full Changelog: v3.2.0...v3.2.1

Release v3.2.0

Feature: Proxy Configuration

In #321 and #325 we implemented four new optional configuration properties that are relevant for users who need to work with proxies:

• proxy_url: The location of your proxy. This could be an http or https url depending on your proxy setup.
• proxy_headers: This param accepts a dictionary which can be used to pass any custom headers required by your proxy. If your proxy is protected by authentication, this is how you can pass basic auth headers with a digest of your username and password.
• ssl_ca_certs: By default the client will perform SSL certificate verification using the CA bundle maintained by Mozilla in the certifi package. If your proxy is using self-signed certs, use this param to specify the path to the certificate (PEM format) we should use to verify your requests.
• ssl_verify: SSL verification is enabled by default, but it can be disabled to aid in troubleshooting.

from pinecone import Pinecone
import urllib3 import make_headers

pc = Pinecone(
    api_key="YOUR_API_KEY",
    proxy_url='https://your-proxy.com',
    proxy_headers=make_headers(proxy_basic_auth='username:password'),
    ssl_ca_certs='path/to/cert-bundle.pem'
)

pc.list_indexes()

Migration from pre-3.0 clients

In older versions of the client you may have passed in proxy configuration to the client like this:

import pinecone
from pinecone.core.client.configuration import OpenApiConfiguration

config = OpenApiConfiguration()
config.ssl_ca_cert = 'path/to/cert-bundle.pem'
config.proxy_headers = make_headers(proxy_basic_auth='username:password')
config.proxy_url = 'https://your-proxy.com'

pinecone.init(
    api_key='your-key',
    environment='us-east1-gcp',
    openapi_config=config
)

If you've tried to pass this openapi_config param into the Pinecone() constructor in the >=3.0 versions of the client, you've run into cryptic bugs telling you your API key is not set (even though it is). We believe we've fixed those bugs in this release, but we strongly encourage you to adopt the new parameters described above. We've decided to move away from having this OpenApiConfiguration as part of our public interface because the object comes from an OpenAPI code generator and has a very large footprint which makes it a challenge from a documentation and testing perspective.

If you were passing configuration through this object for anything not covered by the above four properties, please reach out to let us know and we'll figure out how to handle those configurations going forward.

from pinecone import Pinecone
from pinecone.core.client.configuration import OpenApiConfiguration
import urllib3 import make_headers

config = OpenApiConfiguration()
config.ssl_ca_cert = 'path/to/cert-bundle.pem'
config.proxy_headers = make_headers(proxy_basic_auth='username:password')
config.proxy_url = 'https://your-proxy.com'

pc = Pinecone(
    api_key="YOUR_API_KEY",
    openapi_config=config
) # this works but emits a deprecation notice

Testing

Testing of this feature was done with the help of mitmproxy. You may find it useful to have a look at the tests while troubleshooting your own configuration.

New Contributors

• @GameSetAndMatch made their first contribution in #323

Full Changelog: v3.1.0...v3.2.0

Release v3.1.0

Listing vector ids by prefix in a namespace (for serverless indexes)

We've implemented SDK support for a new data plane endpoint used to list ids by prefix in a given namespace. If the prefix empty string is passed, this can be used to list all ids in a namespace.

The index client now has list and list_paginated. With clever assignment of vector ids, this can be used to help model hierarchical relationships between different vectors such as when you have embeddings for multiple chunks or fragments related to the same document.

The list method returns a generator that handles pagination on your behalf.

from pinecone import Pinecone

pc = Pinecone(api_key='xxx')
index = pc.Index(host='hosturl')

# To iterate over all result pages using a generator function
for ids in index.list(prefix='pref', limit=3, namespace=namespace):
    print(ids) # ['pref1', 'pref2', 'pref3']

    # Now you can pass this id array to other methods, such as fetch or delete.
    vectors = index.fetch(ids=ids, namespace=namespace)

There is also an option to fetch each page of results yourself with list_paginated.

from pinecone import Pinecone

pc = Pinecone(api_key='xxx')
index = pc.Index(host='hosturl')

namespace = 'foo-namespace'

# For manual control over pagination
results = index.list_paginated(
    prefix='pref',
    limit=3,
    namespace='foo',
    pagination_token='eyJza2lwX3Bhc3QiOiI5IiwicHJlZml4IjpudWxsfQ=='
)
print(results.namespace) # 'foo'
print([v.id for v in results.vectors]) # ['pref1', 'pref2', 'pref3']
print(results.pagination.next) # 'eyJza2lwX3Bhc3QiOiI5IiwicHJlZml4IjpudWxsfQ=='
print(results.usage) # { 'read_units': 1 }

Python 3.11 and 3.12 support

We made an adjustment to our declared python version support (from python >=3.8,<3.13 to ^3.8) to make it easier for tools with more expansive statements on what python versions they support to include the pinecone sdk as a dependency. Alongside this change, we expanded our test matrix to include more robust testing with python versions 3.11 and 3.12. Python 3.13 is still in alpha and is not yet part of our test matrix.

• Adjust supported python versions to ^3.8 by @jhamon in #312
• Update pytest-timeout to support python >= 3.12 by @mjvankampen in #314

Chores

• Sync models from pinecone-protos by @fsxfreak in #315
• Fix minor README docs issues in client reference by @austin-denoble in #316

New Contributors

• @mjvankampen made their first contribution in #314

Full Changelog: v3.0.3...v3.1.0

Release v3.0.3

Fixes

• gRPC: parse_query_response: Skip parsing empty Usage by @daverigby in #301
• Support overriding additional_headers with PINECONE_ADDITIONAL_HEADERS environment variable by @fsxfreak in #304
• upsert_from_dataframe: Hide all progressbars if !show_progress by @daverigby in #310

Chores

• Update github actions dependencies to fix warnings by @jhamon in #308
• Update generated openapi code by @jhamon in #309

New Contributors

• @fsxfreak made their first contribution in #304

Full Changelog: v3.0.2...v3.0.3

Release v3.0.2

Fixes

Create indexes using source_collection option in PodSpec

This release resolves a bug when passing source_collection as part of the PodSpec. This option is used when creating a new index from vector data stored in a collection. The value of this field should be a collection you have created previously from an index and that shows with pc.list_collections(). Currently collections and pod-based indexes are not portable across environments.

from pinecone import Pinecone

pc = Pinecone(api_key='YOUR_API_KEY')

pc.create_index(
    name='my-index', 
    dimension=1536, 
    metric='cosine', 
    spec=PodSpec(
        environment='us-east1-gcp', 
        source_collection='collection-2024jan16',
    )
)

Pass optional GRPCClientConfig when using PineconeGRPC

This could be considered as a fix for a UX bug or a micro-feature, depending on your perspective. In 3.0.2 we updated the pc.Index helper method that is used to build instances of the GRPCIndex class. It now accepts an optional keyword param grpc_config. Before this fix, you would need to import GRPCIndex and instantiate GRPCIndex yourself in order to pass this configuration and customize some settings, which was a bit clunky.

from pinecone.grpc import PineconeGRPC, GRPCClientConfig

pc = PineconeGRPC(api_key='YOUR_API_KEY')
grpc_config = GRPCClientConfig(
    timeout=10, 
    secure=True,
    reuse_channel=True
)
index = pc.Index(
   name='my-index', 
   host='host', 
   grpc_config=grpc_config
)

# Now do data operations
index.upsert(...)

Pass optional pool_threads config on the index.

Similar to the grpc_config option, some people requested the ability to pass pool_threads when targeting an index rather than in the initial client initialization. Now the optional configuration is accepted in both places, with the value passed to .Index() taking precedence.

Now these are both valid approaches:

from pinecone import Pinecone

pc = Pinecone(api_key='key', pool_threads=5)
pc.Index(host='host')
pc.upsert(...)

from pinecone import Pinecone

pc = Pinecone(api_key='key')
index = pc.Index(host='host', pool_threads=5)
index.upsert(...)

Debugging

This is probably only relevant for internal Pinecone employees or support agents, but the index client now accepts configuration to attach additional headers to each data plane request. This can help with tracing requests in logs.

from pinecone import Pinecone

pc = Pinecone(api_key='xxx')
index = pc.Index(
    host='hosturl', 
    additional_headers={ 'header-1': 'header-1-value' }
)

# Now do things
index.upsert(...)

The equivalent concept for PineconeGRPC is to pass additional_metadata. gRPC metadata fill a similar role as HTTP request headers, and should not be confused with metadata associated with vectors stored in your Pinecone indexes.

from pinecone.grpc import PineconeGRPC, GRPCClientConfig

pc = PineconeGRPC(api_key='YOUR_API_KEY')

grpc_config = GRPCClientConfig(additional_metadata={'extra-header': 'value123'})
index = pc.Index(
    name='my-index', 
    host='host', 
    grpc_config=grpc_config
)

# do stuff
index.upsert(...)

Changelog

• README.md: Update install steps to escape brackets by @daverigby in #298
• Expose missing configurations for grpc_config and pool_threads by @jhamon in #296
• Integration tests for collections by @jhamon in #299
• Optional configs to pass additional_headers/additional_metadata to indexes by @jhamon in #297

New Contributors

• @daverigby made their first contribution in #298

Full Changelog: v3.0.1...v3.0.2

Release v3.0.1

This is a quick follow-up to the v3.0.0 release earlier this week. This release adds improved error messages to help guide people on how to address some of the breaking changes in v3, such as the migration of core functionality from attributes on the pinecone module into methods of the Pinecone class.

If you're updating from v2.2.x from the first time, you will still want to checkout the v3.0.0 Migration Guide for a walkthrough of all the new features and changes. All of that information is still accurate for this release.