Expose evaluations and metrics (#319)

Makes metrics and evaluations available --------- Co-authored-by: Auto-format Bot <[email protected]>
groundlight · Feb 25, 2025 · 401bd14 · 401bd14
1 parent 0fa6244
commit 401bd14
Show file tree

Hide file tree

Showing 30 changed files with 2,256 additions and 24 deletions.
diff --git a/generated/.openapi-generator/FILES b/generated/.openapi-generator/FILES
@@ -29,6 +29,11 @@ docs/ImageQueriesApi.md
 docs/ImageQuery.md
 docs/ImageQueryTypeEnum.md
 docs/InlineResponse200.md
+docs/InlineResponse2001.md
+docs/InlineResponse2001EvaluationResults.md
+docs/InlineResponse2002.md
+docs/InlineResponse200Summary.md
+docs/InlineResponse200SummaryClassCounts.md
 docs/LabelValue.md
 docs/LabelValueRequest.md
 docs/LabelsApi.md
@@ -98,6 +103,11 @@ groundlight_openapi_client/model/escalation_type_enum.py
 groundlight_openapi_client/model/image_query.py
 groundlight_openapi_client/model/image_query_type_enum.py
 groundlight_openapi_client/model/inline_response200.py
+groundlight_openapi_client/model/inline_response2001.py
+groundlight_openapi_client/model/inline_response2001_evaluation_results.py
+groundlight_openapi_client/model/inline_response2002.py
+groundlight_openapi_client/model/inline_response200_summary.py
+groundlight_openapi_client/model/inline_response200_summary_class_counts.py
 groundlight_openapi_client/model/label_value.py
 groundlight_openapi_client/model/label_value_request.py
 groundlight_openapi_client/model/mode_enum.py
@@ -132,4 +142,6 @@ setup.cfg
 setup.py
 test-requirements.txt
 test/__init__.py
+test/test_payload_template.py
+test/test_payload_template_request.py
 tox.ini
diff --git a/generated/README.md b/generated/README.md
@@ -123,6 +123,8 @@ Class | Method | HTTP request | Description
 *DetectorsApi* | [**create_detector**](docs/DetectorsApi.md#create_detector) | **POST** /v1/detectors | 
 *DetectorsApi* | [**delete_detector**](docs/DetectorsApi.md#delete_detector) | **DELETE** /v1/detectors/{id} | 
 *DetectorsApi* | [**get_detector**](docs/DetectorsApi.md#get_detector) | **GET** /v1/detectors/{id} | 
+*DetectorsApi* | [**get_detector_evaluation**](docs/DetectorsApi.md#get_detector_evaluation) | **GET** /v1/detectors/{id}/evaluation | 
+*DetectorsApi* | [**get_detector_metrics**](docs/DetectorsApi.md#get_detector_metrics) | **GET** /v1/detectors/{detector_id}/metrics | 
 *DetectorsApi* | [**list_detectors**](docs/DetectorsApi.md#list_detectors) | **GET** /v1/detectors | 
 *DetectorsApi* | [**update_detector**](docs/DetectorsApi.md#update_detector) | **PATCH** /v1/detectors/{id} | 
 *EdgeApi* | [**get_model_urls**](docs/EdgeApi.md#get_model_urls) | **GET** /v1/edge/fetch-model-urls/{detector_id}/ | 
@@ -161,6 +163,11 @@ Class | Method | HTTP request | Description
  - [ImageQuery](docs/ImageQuery.md)
  - [ImageQueryTypeEnum](docs/ImageQueryTypeEnum.md)
  - [InlineResponse200](docs/InlineResponse200.md)
+ - [InlineResponse2001](docs/InlineResponse2001.md)
+ - [InlineResponse2001EvaluationResults](docs/InlineResponse2001EvaluationResults.md)
+ - [InlineResponse2002](docs/InlineResponse2002.md)
+ - [InlineResponse200Summary](docs/InlineResponse200Summary.md)
+ - [InlineResponse200SummaryClassCounts](docs/InlineResponse200SummaryClassCounts.md)
  - [LabelValue](docs/LabelValue.md)
  - [LabelValueRequest](docs/LabelValueRequest.md)
  - [ModeEnum](docs/ModeEnum.md)

diff --git a/generated/docs/DetectorsApi.md b/generated/docs/DetectorsApi.md
@@ -7,6 +7,8 @@ Method | HTTP request | Description
 [**create_detector**](DetectorsApi.md#create_detector) | **POST** /v1/detectors | 
 [**delete_detector**](DetectorsApi.md#delete_detector) | **DELETE** /v1/detectors/{id} | 
 [**get_detector**](DetectorsApi.md#get_detector) | **GET** /v1/detectors/{id} | 
+[**get_detector_evaluation**](DetectorsApi.md#get_detector_evaluation) | **GET** /v1/detectors/{id}/evaluation | 
+[**get_detector_metrics**](DetectorsApi.md#get_detector_metrics) | **GET** /v1/detectors/{detector_id}/metrics | 
 [**list_detectors**](DetectorsApi.md#list_detectors) | **GET** /v1/detectors | 
 [**update_detector**](DetectorsApi.md#update_detector) | **PATCH** /v1/detectors/{id} | 
 
@@ -243,6 +245,160 @@ Name | Type | Description  | Notes
  - **Accept**: application/json
 
 
+### HTTP response details
+
+| Status code | Description | Response headers |
+|-------------|-------------|------------------|
+**200** |  |  -  |
+
+[[Back to top]](#) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to Model list]](../README.md#documentation-for-models) [[Back to README]](../README.md)
+
+# **get_detector_evaluation**
+> InlineResponse2001 get_detector_evaluation(id)
+
+
+
+Get Detector evaluation results. The result is null if there isn't enough ground truth data to evaluate the detector. Returns the time of the evaulation, total ground truth labels, the ml based kfold accuracies, and the system accuracies at different confidence thresholds
+
+### Example
+
+* Api Key Authentication (ApiToken):
+
+```python
+import time
+import groundlight_openapi_client
+from groundlight_openapi_client.api import detectors_api
+from groundlight_openapi_client.model.inline_response2001 import InlineResponse2001
+from pprint import pprint
+# Defining the host is optional and defaults to https://api.groundlight.ai/device-api
+# See configuration.py for a list of all supported configuration parameters.
+configuration = groundlight_openapi_client.Configuration(
+    host = "https://api.groundlight.ai/device-api"
+)
+
+# The client must configure the authentication and authorization parameters
+# in accordance with the API server security policy.
+# Examples for each auth method are provided below, use the example that
+# satisfies your auth use case.
+
+# Configure API key authorization: ApiToken
+configuration.api_key['ApiToken'] = 'YOUR_API_KEY'
+
+# Uncomment below to setup prefix (e.g. Bearer) for API key, if needed
+# configuration.api_key_prefix['ApiToken'] = 'Bearer'
+
+# Enter a context with an instance of the API client
+with groundlight_openapi_client.ApiClient(configuration) as api_client:
+    # Create an instance of the API class
+    api_instance = detectors_api.DetectorsApi(api_client)
+    id = "id_example" # str | 
+
+    # example passing only required values which don't have defaults set
+    try:
+        api_response = api_instance.get_detector_evaluation(id)
+        pprint(api_response)
+    except groundlight_openapi_client.ApiException as e:
+        print("Exception when calling DetectorsApi->get_detector_evaluation: %s\n" % e)
+```
+
+
+### Parameters
+
+Name | Type | Description  | Notes
+------------- | ------------- | ------------- | -------------
+ **id** | **str**|  |
+
+### Return type
+
+[**InlineResponse2001**](InlineResponse2001.md)
+
+### Authorization
+
+[ApiToken](../README.md#ApiToken)
+
+### HTTP request headers
+
+ - **Content-Type**: Not defined
+ - **Accept**: application/json
+
+
+### HTTP response details
+
+| Status code | Description | Response headers |
+|-------------|-------------|------------------|
+**200** |  |  -  |
+
+[[Back to top]](#) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to Model list]](../README.md#documentation-for-models) [[Back to README]](../README.md)
+
+# **get_detector_metrics**
+> InlineResponse200 get_detector_metrics(detector_id)
+
+
+
+Get Detector metrics, primarily the counts of different types of labels
+
+### Example
+
+* Api Key Authentication (ApiToken):
+
+```python
+import time
+import groundlight_openapi_client
+from groundlight_openapi_client.api import detectors_api
+from groundlight_openapi_client.model.inline_response200 import InlineResponse200
+from pprint import pprint
+# Defining the host is optional and defaults to https://api.groundlight.ai/device-api
+# See configuration.py for a list of all supported configuration parameters.
+configuration = groundlight_openapi_client.Configuration(
+    host = "https://api.groundlight.ai/device-api"
+)
+
+# The client must configure the authentication and authorization parameters
+# in accordance with the API server security policy.
+# Examples for each auth method are provided below, use the example that
+# satisfies your auth use case.
+
+# Configure API key authorization: ApiToken
+configuration.api_key['ApiToken'] = 'YOUR_API_KEY'
+
+# Uncomment below to setup prefix (e.g. Bearer) for API key, if needed
+# configuration.api_key_prefix['ApiToken'] = 'Bearer'
+
+# Enter a context with an instance of the API client
+with groundlight_openapi_client.ApiClient(configuration) as api_client:
+    # Create an instance of the API class
+    api_instance = detectors_api.DetectorsApi(api_client)
+    detector_id = "detector_id_example" # str | 
+
+    # example passing only required values which don't have defaults set
+    try:
+        api_response = api_instance.get_detector_metrics(detector_id)
+        pprint(api_response)
+    except groundlight_openapi_client.ApiException as e:
+        print("Exception when calling DetectorsApi->get_detector_metrics: %s\n" % e)
+```
+
+
+### Parameters
+
+Name | Type | Description  | Notes
+------------- | ------------- | ------------- | -------------
+ **detector_id** | **str**|  |
+
+### Return type
+
+[**InlineResponse200**](InlineResponse200.md)
+
+### Authorization
+
+[ApiToken](../README.md#ApiToken)
+
+### HTTP request headers
+
+ - **Content-Type**: Not defined
+ - **Accept**: application/json
+
+
 ### HTTP response details
 
 | Status code | Description | Response headers |

diff --git a/generated/docs/InlineResponse200.md b/generated/docs/InlineResponse200.md
@@ -4,7 +4,7 @@
 ## Properties
 Name | Type | Description | Notes
 ------------ | ------------- | ------------- | -------------
-**username** | **str** | The user&#39;s username | [optional] 
+**summary** | [**InlineResponse200Summary**](InlineResponse200Summary.md) |  | [optional] 
 **any string name** | **bool, date, datetime, dict, float, int, list, str, none_type** | any string name can be used but the value must be the correct type | [optional]
 
 [[Back to Model list]](../README.md#documentation-for-models) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to README]](../README.md)

diff --git a/generated/docs/InlineResponse2001.md b/generated/docs/InlineResponse2001.md
@@ -0,0 +1,12 @@
+# InlineResponse2001
+
+
+## Properties
+Name | Type | Description | Notes
+------------ | ------------- | ------------- | -------------
+**evaluation_results** | [**InlineResponse2001EvaluationResults**](InlineResponse2001EvaluationResults.md) |  | [optional] 
+**any string name** | **bool, date, datetime, dict, float, int, list, str, none_type** | any string name can be used but the value must be the correct type | [optional]
+
+[[Back to Model list]](../README.md#documentation-for-models) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to README]](../README.md)
+
+
diff --git a/generated/docs/InlineResponse2001EvaluationResults.md b/generated/docs/InlineResponse2001EvaluationResults.md
@@ -0,0 +1,19 @@
+# InlineResponse2001EvaluationResults
+
+
+## Properties
+Name | Type | Description | Notes
+------------ | ------------- | ------------- | -------------
+**eval_timestamp** | **datetime** |  | [optional] 
+**total_ground_truth_examples** | **int, none_type** |  | [optional] 
+**kfold_pooled__balanced_accuracy** | **float** |  | [optional] 
+**kfold_pooled__positive_accuracy** | **float** |  | [optional] 
+**kfold_pooled__negative_accuracy** | **float** |  | [optional] 
+**balanced_system_accuracies** | **{str: (bool, date, datetime, dict, float, int, list, str, none_type)}** |  | [optional] 
+**positive_system_accuracies** | **{str: (bool, date, datetime, dict, float, int, list, str, none_type)}** |  | [optional] 
+**negative_system_accuracies** | **{str: (bool, date, datetime, dict, float, int, list, str, none_type)}** |  | [optional] 
+**any string name** | **bool, date, datetime, dict, float, int, list, str, none_type** | any string name can be used but the value must be the correct type | [optional]
+
+[[Back to Model list]](../README.md#documentation-for-models) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to README]](../README.md)
+
+
diff --git a/generated/docs/InlineResponse2002.md b/generated/docs/InlineResponse2002.md
@@ -0,0 +1,12 @@
+# InlineResponse2002
+
+
+## Properties
+Name | Type | Description | Notes
+------------ | ------------- | ------------- | -------------
+**username** | **str** | The user&#39;s username | [optional] 
+**any string name** | **bool, date, datetime, dict, float, int, list, str, none_type** | any string name can be used but the value must be the correct type | [optional]
+
+[[Back to Model list]](../README.md#documentation-for-models) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to README]](../README.md)
+
+
diff --git a/generated/docs/InlineResponse200Summary.md b/generated/docs/InlineResponse200Summary.md
@@ -0,0 +1,16 @@
+# InlineResponse200Summary
+
+
+## Properties
+Name | Type | Description | Notes
+------------ | ------------- | ------------- | -------------
+**num_ground_truth** | **int** |  | [optional] 
+**num_current_source_human** | **int** |  | [optional] 
+**class_counts** | [**InlineResponse200SummaryClassCounts**](InlineResponse200SummaryClassCounts.md) |  | [optional] 
+**unconfident_counts** | **{str: (bool, date, datetime, dict, float, int, list, str, none_type)}** |  | [optional] 
+**total_iqs** | **int** |  | [optional] 
+**any string name** | **bool, date, datetime, dict, float, int, list, str, none_type** | any string name can be used but the value must be the correct type | [optional]
+
+[[Back to Model list]](../README.md#documentation-for-models) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to README]](../README.md)
+
+
diff --git a/generated/docs/InlineResponse200SummaryClassCounts.md b/generated/docs/InlineResponse200SummaryClassCounts.md
@@ -0,0 +1,16 @@
+# InlineResponse200SummaryClassCounts
+
+
+## Properties
+Name | Type | Description | Notes
+------------ | ------------- | ------------- | -------------
+**source_ml** | **{str: (bool, date, datetime, dict, float, int, list, str, none_type)}** |  | [optional] 
+**source_human** | **{str: (bool, date, datetime, dict, float, int, list, str, none_type)}** |  | [optional] 
+**cloud_labeler** | **{str: (bool, date, datetime, dict, float, int, list, str, none_type)}** |  | [optional] 
+**cloud** | **{str: (bool, date, datetime, dict, float, int, list, str, none_type)}** |  | [optional] 
+**total** | **{str: (bool, date, datetime, dict, float, int, list, str, none_type)}** |  | [optional] 
+**any string name** | **bool, date, datetime, dict, float, int, list, str, none_type** | any string name can be used but the value must be the correct type | [optional]
+
+[[Back to Model list]](../README.md#documentation-for-models) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to README]](../README.md)
+
+
diff --git a/generated/docs/TextRecognitionResult.md b/generated/docs/TextRecognitionResult.md
@@ -4,7 +4,7 @@
 ## Properties
 Name | Type | Description | Notes
 ------------ | ------------- | ------------- | -------------
-**text** | **str, none_type** |  | 
+**text** | **str** |  | 
 **truncated** | **bool** |  | 
 **confidence** | **float, none_type** |  | [optional] 
 **source** | **str** |  | [optional] 

diff --git a/generated/docs/UserApi.md b/generated/docs/UserApi.md
@@ -8,7 +8,7 @@ Method | HTTP request | Description
 
 
 # **who_am_i**
-> InlineResponse200 who_am_i()
+> InlineResponse2002 who_am_i()
 
 
 
@@ -22,7 +22,7 @@ Retrieve the current user.
 import time
 import groundlight_openapi_client
 from groundlight_openapi_client.api import user_api
-from groundlight_openapi_client.model.inline_response200 import InlineResponse200
+from groundlight_openapi_client.model.inline_response2002 import InlineResponse2002
 from pprint import pprint
 # Defining the host is optional and defaults to https://api.groundlight.ai/device-api
 # See configuration.py for a list of all supported configuration parameters.
@@ -60,7 +60,7 @@ This endpoint does not need any parameter.
 
 ### Return type
 
-[**InlineResponse200**](InlineResponse200.md)
+[**InlineResponse2002**](InlineResponse2002.md)
 
 ### Authorization