Move all grpc stuff into pinecone.grpc subpackage

[jhamon]

Problem

In the past this package attempted to import extra grpc deps in the root __init__.py inside a try/catch that caught ImportError when deps were missing. This is error prone because people may be attempting to use the pinecone module alongside other packages that depend on incompatible versions of these dependencies, so we shouldn't interpret successful import as a sign that a user has installed the pinecone-client[grpc] extras (with the versions we expect). When these incompatible versions are present, the ImportError doesn't happen but other runtime errors can occur even when the person is not attempting to use GRPCIndex.

Solution

The gist of this PR is to better isolate everything GRPC-related into a subpackage to minimize the changes of a grpc-related dependency clash getting in the way of using Pinecone. By doing this, issues with grpc dependencies should never impact people using the non-grpc client.

Summary of changes:

• Moving files
  • Contents of pinecone/data/grpc moved up and over to pinecone/grpc
  • Tests moved from tests/unit/test_index_grpc.py into tests/unit_grpc/test_index_grpc.py. This is to make it easier to split grpc and non-grpc test runs.
  • Delete remaining cruft in pinecone/deprecated/legacy_utils.py because they were unused and loading grpc deps.
• Making from pinecone import Pinecone independent of GRPC
  • References to all GRPC deps removed from pinecone/__init__.py
  • Adjusting pinecone/pinecone.py to remove IndexGRPC() method.
• Setting up for from pinecone.grpc import Pinecone
  • Add pinecone/grpc/pinecone.yaml that extends the other Pinecone class, modifying only the Index() method to use GRPCIndex.
  • Add pinecone/grpc/__init__.py with entries for grpc-flavored Pinecone and IndexGRPC exports.
• CI updates
  • pyproject.toml adjusted to properly indicate grpc deps as optional = true; I discovered in testing that grpc dependencies were always being installed in CI.
  • Add an input var to the setup-poetry action so that we can toggle on grpc dependencies when needed.
  • .github/workflows/testing.yaml refactored to break GRPC and non-GRPC into separate jobs, since they have different install requirements.

Usage evolution

Legacy implementation

import pinecone 
pinecone.init(api_key='foo', environment='bar')

index = pinecone.Index('index-name')
index_grpc = pinecone.GRPCIndex('index-name')

Unreleased spruce branch (before this PR)

This was an improvement over the legacy approach because we no longer relied on global state for configuration. Instead, we wrap configuration state into a class instance. However, the Pinecone class had a dependency on GRPC stuff to implement the GRPCIndex() method. The imports of these extra deps was intended to noop when they were not installed, but as I described above there are situations where you could still have runtime errors if other dependencies in your notebook / app have installed incompatible GRPC dependencies that prevent you from using the REST version of the client.

from pinecone import Pinecone

p = Pinecone(api_key='foo')
index = p.Index('index-name')
index_grpc = p.GRPCIndex('index-name')

After this PR, import from different subpackage when GRPC desired

Now the usage is exactly the same for REST vs GRPC except for the import coming from pinecone.grpc instead of pinecone when you want data operations to use grpc. GRPC deps are not loaded at all unless you specifically import from pinecone.grpc.

To use REST, import from pinecone

from pinecone import Pinecone

p = Pinecone(api_key='foo')
index = p.Index('index-name')

To use GRPC, import from pinecone.grpc subpackage

from pinecone.grpc import Pinecone

p = Pinecone(api_key='foo')
index = p.Index('index-name')

If you ever wanted to use these side by side for some reason (no real reason to ever do this unless you are writing tests for this package) you would have to alias one of the imports to avoid a name collision.

from pinecone import Pinecone
from pinecone.grpc import Pinecone as PineconeGRPC

...

Type of Change

• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update
• Infrastructure change (CI configs, etc)

Test Plan

Describe specific steps for validating this change.

[austin-denoble]

Nice work, all of this makes a lot of sense. Thanks for taking this through the finish line after helping out with all the dependency problems that cropped up due to this setup. 🚢

[austin-denoble]

Thanks for pulling this out of here, this was a mistake on my part I think.

We cover this step in the pr workflow already so it may have been running twice.

[austin-denoble]

Did not know you could explicitly set optional like this. 👍