Skip to content

Commit

Permalink
Parameterize integration tests on environment (#256)
Browse files Browse the repository at this point in the history
## Problem

I want to be able to run the integration tests with a variety of
environments.

## Solution

Add some variables to the test matrix and wire them into the test
execution. We can expand this list over time to grow our test surface
area.

## Type of Change

- [x] Infrastructure change (CI configs, etc)
  • Loading branch information
jhamon authored Dec 26, 2023
1 parent 3792139 commit bcf5393
Show file tree
Hide file tree
Showing 3 changed files with 33 additions and 8 deletions.
18 changes: 15 additions & 3 deletions .github/workflows/testing.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -37,19 +37,25 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: [3.8, 3.11]
pineconeEnv:
- prod
# - staging
testConfig:
- python-version: 3.8 # Do one test run with 3.8 for sanity check
pod: { environment: 'us-east1-gcp'}
serverless: { cloud: 'aws', region: 'us-west-2'}
- python-version: 3.11
pod: { environment: 'us-east1-gcp'}
serverless: { cloud: 'aws', region: 'us-west-2'}
max-parallel: 1
fail-fast: false
steps:
- uses: actions/checkout@v4

- name: Set up Python ${{ matrix.python-version }}
- name: Set up Python ${{ matrix.testConfig.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
python-version: ${{ matrix.testConfig.python-version }}

- name: Setup Poetry
uses: ./.github/actions/setup-poetry
Expand All @@ -61,6 +67,9 @@ jobs:
PINECONE_CONTROLLER_HOST: 'https://api.pinecone.io'
PINECONE_API_KEY: ${{ secrets.PINECONE_API_KEY }}
GITHUB_BUILD_NUMBER: "${{ github.run_number }}-p-${{ matrix.python-version}}"
POD_ENVIRONMENT: ${{ matrix.testConfig.pod.environment }}
SERVERLESS_CLOUD: ${{ matrix.testConfig.serverless.cloud }}
SERVERLESS_REGION: ${{ matrix.testConfig.serverless.region }}

- name: Run integration tests (REST, staging)
if: matrix.pineconeEnv == 'staging'
Expand All @@ -69,6 +78,9 @@ jobs:
PINECONE_CONTROLLER_HOST: 'https://api-staging.pinecone.io'
PINECONE_API_KEY: ${{ secrets.PINECONE_API_KEY_STAGING }}
GITHUB_BUILD_NUMBER: "${{ github.run_number }}-s-${{ matrix.python-version}}"
POD_ENVIRONMENT: ${{ matrix.testConfig.pod.environment }}
SERVERLESS_CLOUD: ${{ matrix.testConfig.serverless.cloud }}
SERVERLESS_REGION: ${{ matrix.testConfig.serverless.region }}

units-grpc:
name: Unit tests (GRPC)
Expand Down
18 changes: 15 additions & 3 deletions tests/integration/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,27 @@ def client():
api_key = get_environment_var('PINECONE_API_KEY')
return Pinecone(api_key=api_key)

@pytest.fixture()
def pod_environment():
return get_environment_var('POD_ENVIRONMENT', 'us-east1-gcp')

@pytest.fixture()
def serverless_cloud():
return get_environment_var('SERVERLESS_CLOUD', 'aws')

@pytest.fixture()
def serverless_region():
return get_environment_var('SERVERLESS_REGION', 'us-west-2')

@pytest.fixture()
def environment():
return get_environment_var('PINECONE_ENVIRONMENT')

@pytest.fixture()
def create_sl_index_params(index_name):
def create_sl_index_params(index_name, serverless_cloud, serverless_region):
spec = {"serverless": {
'cloud': 'aws',
'region': 'us-west-2'
'cloud': serverless_cloud,
'region': serverless_region
}}
return dict(name=index_name, dimension=10, metric='cosine', spec=spec)

Expand Down
5 changes: 3 additions & 2 deletions tests/integration/helpers/helpers.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
import os
import random
import string
from typing import Any

def random_string(length):
return ''.join(random.choice(string.ascii_lowercase) for i in range(length))
Expand Down Expand Up @@ -37,8 +38,8 @@ def generate_index_name(test_name: str) -> str:

return index_name.lower()

def get_environment_var(name: str) -> str:
val = os.getenv(name)
def get_environment_var(name: str, defaultVal: Any = None) -> str:
val = os.getenv(name, defaultVal)
if (val is None):
raise Exception('Expected environment variable ' + name + ' is not set')
else:
Expand Down

0 comments on commit bcf5393

Please sign in to comment.