Wire up plugin interface

[jhamon]

Problem

We need a way to develop, test, demo, version and release features when they are in a preview or pre-release state independent from the stable core sdk. New features should only enter the sdk core when rapid refinement and iteration has tapered off and we're ready to stabilize the feature. This will help us innovate and ship faster.

Solution

We have implemented a plugin interface that allows us to extend the Pinecone class with experimental features defined elsewhere. For now, this plugin interface is only intended for internal use at Pinecone.

There are three pieces involved here: the changes in this diff, a small separate package called pinecone_plugin_interface, and the plugin implementations themself. Currently, there are no publicly available plugins.

The Pinecone class here imports a load_and_install function from the pinecone_plugin_interface package. When invoked, this function scans the user's environment looking for packages that have been implemented within a namespace package called pinecone_plugins. If it finds any, it will use the client_builder function supplied by the Pinecone class to pass user configuration into the plugin for consistent handling of configurations such as api keys, proxy settings, etc.

Although the amount of code involved in pinecone_plugin_interface is quite meager, externalizing it should give some flexibility to share code between the sdk and plugin implementations without creating a circular dependency should we ever wish to include a plugin as a dependency of the sdk (after functionality has stabilized).

I have been defensive here by wrapping the plugin installation in a try/catch. I want to be confident that the mere presence of an invalid or broken plugin cannot break the rest of the sdk.

Type of Change

• New feature (non-breaking change which adds functionality)

Test Plan

Tests should be green. I added some unit tests covering the new builder function stuff. As far as integration testing, I can't really add those here until we have publicly available plugins. But I've done manual testing to gain confidence.

[austin-denoble]

Great work, Jen. I need to spend more time getting familiar with the https://github.com/pinecone-io/python-plugin-interface package, but the changes to the client surface are really straightforward.

[austin-denoble]

super nit: I think there's an extra period after interchangeably. Thank you for these detailed comments also, helpful for putting the pieces together.