Basic python SDK (#2)

* A small wrapper layer around the auto-generated sdk

* Update api

* Fix url

* Change to pip/poetry install

* Add github action for publishing on releases

* Use v0.1.2-rc1 to test publishing

* 0.1.2

* Add instructions on releases

* Improve readme

* simple_example

* readme

* readme

* more readme

* Apply patch; add tests

* Use alternative python

* files

* Better client with datamodel and working submit_image_query

* IMprove interface with types

* Examples

* Better test-local and test-integ

* Add github action to run integration tests

* Add pillow

* Bump to version 0.1.3

* Some cleanup

* Address feedback

.github/workflows/publish.yaml

@@ -0,0 +1,36 @@
+# When a release is created on github, publish the groundlight package to our internal pypi repository
+# (Similar to https://github.com/positronix-ai/predictors/blob/main/.github/workflows/publish.yaml)
+name: publish package
+on:
+  release:
+    types: [created]
+jobs:
+  publish-python-package:
+    runs-on: ubuntu-latest
+    env:
+      INTERNAL_REPO_URL: https://positronix-723181461334.d.codeartifact.us-west-2.amazonaws.com/pypi/internal/
+    steps:
+      - name: install python
+        uses: actions/setup-python@v2
+        with:
+          python-version: 3.9
+      - name: install poetry
+        run: |
+          pip install -U pip
+          pip install poetry
+      - name: configure AWS credentials
+        uses: aws-actions/configure-aws-credentials@v1
+        with:
+          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
+          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+          aws-region: us-west-2
+      - name: get code
+        uses: actions/checkout@v2
+      - name: build package
+        run: poetry build
+      - name: configure poetry and publish
+        run: |
+          export POETRY_HTTP_BASIC_INTERNAL_USERNAME=aws
+          export POETRY_HTTP_BASIC_INTERNAL_PASSWORD=$(aws codeartifact get-authorization-token --domain positronix --query authorizationToken --output text)
+          poetry config repositories.internal $INTERNAL_REPO_URL
+          poetry publish -r internal

.github/workflows/test-integ.yaml

@@ -0,0 +1,23 @@
+# Run integration tests against the integ API endpoint
+name: test integ
+on: [push]
+jobs:
+  run-tests:
+    runs-on: ubuntu-latest
+    env:
+      # This is associated with the "sdk-integ-test" user, credentials on 1password
+      GROUNDLIGHT_API_TOKEN: ${{ secrets.GROUNDLIGHT_API_TOKEN }}
+    steps:
+      - name: get code
+        uses: actions/checkout@v2
+      - name: install python
+        uses: actions/setup-python@v2
+        with:
+          python-version: 3.9
+      - name: install poetry
+        run: |
+          pip install -U pip
+          pip install poetry
+          poetry install
+      - name: run tests
+        run: make test-integ

.gitignore

@@ -158,3 +158,6 @@ cython_debug/
 #  and can be added to the global gitignore or merged into this file.  For a more nuclear
 #  option (not recommended) you can uncomment the following to ignore the entire idea folder.
 #.idea/
+
+# This is a library, so we want clients to have more flexible dependencies. So we don't include a lockfile.
+poetry.lock

Makefile

@@ -1,8 +1,20 @@
+.PHONY: all test clean
+
+install:  ## Install the package from source
+	poetry install
+
 # Java weirdness - see https://github.com/OpenAPITools/openapi-generator/issues/11763#issuecomment-1098337960
-generate:  ## Generate the SDK from our public openapi spec
+generate: install  ## Generate the SDK from our public openapi spec
 	_JAVA_OPTIONS="--add-opens=java.base/java.lang=ALL-UNNAMED \
 	--add-opens=java.base/java.util=ALL-UNNAMED" \
 		openapi-generator generate -i spec/public-api.yaml \
 		-g python \
 		-o ./generated
 	poetry run datamodel-codegen  --input spec/public-api.yaml --output generated/model.py
+
+test-local: install  ## Run integration tests against an API server running at http://localhost:8000/device-api (needs GROUNDLIGHT_API_TOKEN)
+	GROUNDLIGHT_TEST_API_ENDPOINT="http://localhost:8000/device-api" poetry run pytest --cov=src test --log-cli-level INFO
+
+
+test-integ: install  ## Run integration tests against the integ API server (needs GROUNDLIGHT_API_TOKEN)
+	GROUNDLIGHT_TEST_API_ENDPOINT="https://device.integ.positronix.ai/device-api" poetry run pytest --cov=src test --log-cli-level INFO

README.md

@@ -1 +1,117 @@
 # Groundlight Python SDK
+
+This package holds an SDK for accessing the Groundlight public API. 
+
+### Installation
+
+[Make sure you have internal pypi credentials set up](https://github.com/positronix-ai/tools/blob/main/internal-pip-login.sh), and then install with `pip` or `poetry`.
+
+```Bash
+# pip
+$ pip install groundlight
+
+# poetry
+$ poetry add groundlight
+```
+
+### Basic Usage
+
+To access the API, you need an API token. You can create one at [app.groundlight.ai](https://app.positronix.ai/reef/my-account/api-tokens). Then, add it as an environment variable called `GROUNDLIGHT_API_TOKEN`:
+
+```Bash
+$ export GROUNDLIGHT_API_TOKEN=tok_abc123
+```
+
+Now you can use the python SDK!
+
+```Python
+from groundlight import Groundlight
+
+# Load the API client. This defaults to the prod endpoint,
+# but you can specify a different endpoint like so:
+# gl = Groundlight(endpoint="https://device.integ.positronix.ai/device-api")
+gl = Groundlight()
+
+# Call an API method (e.g., retrieve a list of detectors)
+detectors = gl.list_detectors()
+```
+
+### What API methods are available?
+
+Check out the [User Guide](UserGuide.md)!
+
+For more details, see the [Groundlight](src/groundlight/client.py) class. This SDK closely follows the methods in our [API Docs](https://app.positronix.ai/reef/admin/api-docs).
+
+
+## Development
+
+The auto-generated SDK code is in the `generated/` directory. To re-generate the client code, you'll need to install [openapi-generator](https://openapi-generator.tech/docs/installation#homebrew) (I recommend homebrew if you're on a mac). Then you can run it with:
+
+```Bash
+$ make generate
+```
+
+## Testing
+
+Most tests need an API endpoint to run.
+
+### Local API endpoint
+
+1. Set up a local [janzu API endpoint](https://github.com/positronix-ai/zuuul/blob/main/deploy/README.md#development-using-local-microk8s) running (e.g., on an AWS GPU instance).
+
+1. Set up an ssh tunnel to your laptop. That way, you can access the endpoint at http://localhost:8000/device-api (and the web UI at http://localhost:8000/reef):
+
+    ```Bash
+    $ ssh instance-name -L 8000:localhost:80
+    ```
+
+1. Run the tests (with an API token)
+
+    ```Bash
+    $ export GROUNDLIGHT_API_TOKEN=tok_abc123
+    $ make test-local
+    ```
+
+(Note: in theory, it's possible to run the janzu API server on your laptop without microk8s - but some API methods don't work because of the dependence on GPUs)
+
+### Integ API endpoint
+
+1. Run the tests (with an API token)
+
+    ```Bash
+    $ export GROUNDLIGHT_API_TOKEN=tok_abc123
+    $ make test-integ
+    ```
+
+## Releases
+
+To publish a new package version to our [internal pypi repository](https://github.com/positronix-ai/packaging/tree/main/aws), you create a release on github.
+
+```Bash
+# Create a git tag locally. Use semver "vX.Y.Z" format.
+$ git tag -a v0.1.2 -m "Short description"
+
+# Push the tag to the github repo
+$ git push origin --tags
+```
+
+Then, go to the [github repo](https://github.com/positronix-ai/groundlight-python-sdk/tags) -> choose your tag -> create a release from this tag -> type in some description -> release. A [github action](https://github.com/positronix-ai/groundlight-python-sdk/actions/workflows/publish.yaml) will trigger a release, and then `groundlight-X.Y.Z` will be available for consumers.
+
+## TODOs
+
+- Figure out how we want to handle tests (since almost everything is an integration test). And, running the stateful (creation) tests can lead to a bunch of objects in the DB.
+- Improve wrappers around API functions (e.g., simplify the responses even further, add auto-pagination managers, etc.)
+  - The SDK should allow you to work with the most natural interface, rather than trying to exactly mirror the REST API.
+  - E.g.
+    - Add an image query long polling helper method (calls POST, then several GETs)
+    - It would be nice to have a `get_or_create_detector()` function (even better if it's supported in the API directly). That way, "submit image query" code examples will be simpler.
+- Better auto-generated code docs (e.g. [sphinx](https://www.sphinx-doc.org/en/master/))
+  - Model types (e.g., [autodoc_pydantic](https://github.com/mansenfranzen/autodoc_pydantic))
+  - Cleaner auto-generated model names (e.g., `PaginatedDetectorList` is a little ugly)
+- Better versioning strategy. On the one hand, this package will closely follow the versioning in the HTTP API. On the other hand, we may add features in the client (like image utils, shortcuts, etc.) that are not in the REST API.
+- Better way of managing dependency on `public-api.yaml` OpenAPI spec (right now, we just copy the file over manually)
+- Update the web links (links to website, link to API endpoint, etc.)
+- `with` context manager (auto cleanup the client object)
+- It would be great to add notebooks with interactive examples that can actually run out of the box
+- Have a cleaner distinction between dev docs and user guide docs
+- ...

UserGuide.md

@@ -0,0 +1,114 @@
+# User Guide
+
+## Pre-reqs
+
+For all the examples, there are 3 pre-reqs:
+
+1. [Make sure you have internal pypi credentials set up](https://github.com/positronix-ai/tools/blob/main/internal-pip-login.sh), and then install with `pip` or `poetry`.
+
+    ```Bash
+    # pip
+    $ pip install groundlight
+
+    # poetry
+    $ poetry add groundlight
+    ```
+
+1. To access the API, you need an API token. You can create one at [app.groundlight.ai](https://app.positronix.ai/reef/my-account/api-tokens). Then, add it as an environment variable called `GROUNDLIGHT_API_TOKEN`:
+
+    ```Bash
+    $ export GROUNDLIGHT_API_TOKEN=tok_abc123
+    ```
+
+1. Create the `Groundlight` API client. We usually use `gl` as a shorthand name, but you are free to name it what you like!
+
+    ```Python
+    from groundlight import Groundlight
+    gl = Groundlight()
+    ```
+## Basics
+
+#### Create a new detector
+
+```Python
+detector = gl.create_detector(name="Dog", query="Is it a dog?")
+```
+
+#### Retrieve a detector
+
+```Python
+detector = gl.get_detector(id="YOUR_DETECTOR_ID")
+```
+
+#### List your detectors
+
+```Python
+# Defaults to 10 results per page
+detectors = gl.list_detectors()
+
+# Pagination: 3rd page of 25 results per page
+detectors = gl.list_detectors(page=3, page_size=25)
+```
+
+#### Submit an image query
+
+```Python
+image_query = gl.submit_image_query(detector_id="YOUR_DETECTOR_ID", image="path/to/filename.jpeg")
+```
+
+#### Retrieve an image query
+
+In practice, you may want to check for a new result on your query. For example, after a cloud reviewer labels your query. For example, you can use the `image_query.id` after the above `submit_image_query()` call.
+
+```Python
+image_query = gl.get_image_query(id="YOUR_IMAGE_QUERY_ID")
+```
+
+#### List your previous image queries
+
+```Python
+# Defaults to 10 results per page
+image_queries = gl.list_image_queries()
+
+# Pagination: 3rd page of 25 results per page
+image_queries = gl.list_image_queries(page=3, page_size=25)
+```
+
+## Advanced
+
+#### Use a different API endpoint
+
+```Python
+from groundlight import Groundlight
+
+# Integ
+integ_gl = Groundlight(endpoint="https://device.integ.positronix.ai/device-api")
+
+# Local
+local_gl = Groundlight(endpoint="http://localhost:8000/device-api")
+```
+
+#### Do more with the object models
+
+You can see the different model types [here](generated/model.py). (TODO: Use something like [autodoc_pydantic](https://github.com/mansenfranzen/autodoc_pydantic) to create docs).
+
+All of the `Groundlight` methods return [pydantic](https://pydantic-docs.helpmanual.io/) models - `Detector`, `ImageQuery`, `PaginatedDetectorList`, etc. This provides several benefits: you can access model fields with dot notation, get auto-complete in your IDE, have `model.dict()`, `model.json()`, `model.pickle()` serializers, etc. See more on the [pydantic docs](https://pydantic-docs.helpmanual.io/usage/models/).
+
+### Handling HTTP errors
+
+If there is an HTTP error during an API call, it will raise an `ApiException`. You can access different metadata from that exception:
+
+```Python
+from groundlight import ApiException, Groundlight
+
+gl = Groundlight()
+try:
+    detectors = gl.list_detectors()
+except ApiException as e:
+    print(e)
+    print(e.args)
+    print(e.body)
+    print(e.headers)
+    print(e.reason)
+    print(e.status)
+```

bin/simple_example.py

@@ -0,0 +1,5 @@
+from groundlight import Groundlight
+
+gl = Groundlight()
+detectors = gl.list_detectors()
+print(f"Found {detectors.count} detectors")

pyproject.toml

@@ -0,0 +1,32 @@
+[tool.poetry]
+name = "groundlight"
+version = "0.1.4"
+description = "Call the Groundlight API from python"
+authors = ["Groundlight AI <[email protected]>"]
+packages = [
+    { include = "**/*.py", from = "src" },
+    { include = "**/*.py", from = "generated" },
+]
+
+[tool.poetry.dependencies]
+python = "^3.9"
+python-dateutil = "^2.8.2"
+urllib3 = "^1.26.9"
+frozendict = "^2.3.2"
+certifi = "^2021.10.8"
+pydantic = "^1.9.0"
+
+[tool.poetry.dev-dependencies]
+pytest-cov = "^3.0.0"
+pylint = "^2.13.8"
+black = "^22.3.0"
+flake8 = "^4.0.1"
+mypy = "^0.950"
+pytest = "^7.1.2"
+isort = "^5.10.1"
+autoflake = "^1.4"
+datamodel-code-generator = "^0.12.0"
+
+[build-system]
+requires = ["poetry-core>=1.0.0"]
+build-backend = "poetry.core.masonry.api"

src/groundlight/__init__.py

@@ -0,0 +1,6 @@
+# flake8: noqa
+# Add useful imports from the generated code here at the top level, as a convenience.
+from openapi_client import ApiException
+
+# Imports from our code
+from .client import Groundlight