Feature Request: Map Search Results to Python Types

[rob-buskens-sp]

v3 Search API results are changed from defined types to an object.
This makes working with search results more difficult in some ways. The results have to be treated as a list of dicts.
Predominantly, the APIs use defined types.

So looping through search results:

import sailpoint
import sailpoint.v3
import sailpoint.beta
import logging

from sailpoint.configuration import Configuration
from sailpoint.paginator import Paginator
from sailpoint.v3.models.search import Search
from pprint import pprint
from urllib3 import Retry

# from http.client import HTTPConnection

if __name__ == '__main__':
    logging.basicConfig(filename='audit.log',
                        filemode='a',
                        format='%(asctime)s,%(msecs)d %(name)s %(levelname)s %(message)s',
                        datefmt='%H:%M:%S',
                        level=getattr(logging, "DEBUG"))
                        
    # HTTPConnection.debuglevel = 1 # Low level, goes to stdout
    # Requests logging
    requests_log = logging.getLogger("requests.packages.urllib3")
    requests_log.setLevel('DEBUG')
    requests_log.propagate = True

    configuration = Configuration()
    configuration.retries = Retry(
            total = 5,
            respect_retry_after_header = True
        )

    with sailpoint.v3.ApiClient(configuration) as v3_client, \
        sailpoint.beta.ApiClient(configuration) as beta_client:

        search = sailpoint.v3.Search(
            indices=['events'], 
            query=sailpoint.v3.Query(query='type:*'),
            sort=['-created'])
        
        events = Paginator.paginate_search(sailpoint.v3.SearchApi(v3_client), search=search, increment=100, limit=1000)

        for event in events:
            print(f'{event["created"]:<20} {event["name"]:<20} {event["actor"]["name"]:<20} {event["target"]["name"]:<20}')

Note in the last printf how it's event["created"] rather than event.created

I see a build step in https://github.com/sailpoint-oss/python-sdk/blob/main/sdk-resources/prescript.js that changes the type.

might be an advantage to being able to request how they are returned or have a different api.

feature ask is to be able to get typed return data.

[rob-buskens-sp]

looks like it may take some work to put in type determination. I took out the change to object type. seems the generated code defaults to a type attribute which probably isn't there and it didn't know what type it was. I may have a closer look at this to see if some soft of 'hints' can be put in.

Exception has occurred: ValueError
Multiple matches found when deserializing the JSON string into SearchDocument with oneOf schemas: AccessProfileDocument, AccountActivityDocument, EntitlementDocument, EventDocument, IdentityDocument, RoleDocument. Details: 2 validation errors for EventDocument

[rob-buskens-sp]

The plot thickens... there is a _type on events anyway.

 {
        "org": "company9208-poc",
        "pod": "se01-useast1",
        "created": "2024-03-23T03:03:17.902Z",
        "id": "0043afa0555435216a72d0c797c63d2284d5bfedec2636b3c61b5ad4c0befd18",
        "action": "delete",
        "type": "PROVISIONING",
        "actor": {
            "name": "unknown"
        },
        "target": {
            "name": "[email protected]"
        },
        "stack": "sweep",
        "trackingNumber": "e23c20014dea4654aa6263c460859eba",
        "attributes": {
            "sourceName": "null"
        },
        "objects": [
            "IDENTITY"
        ],
        "operation": "DELETE",
        "status": "PASSED",
        "technicalName": "IDENTITY_DELETE_PASSED",
        "name": "Delete Identity Passed",
        "synced": "2024-03-23T03:03:18.505Z",
        "_type": "event",
        "_version": "v7"
    },

[rob-buskens-sp]

Progress

Some progress as the following shows SearchDocuments are typed and the EventDocument attributes can be reference using .actor.name notation instead of subscript ["actor"]["name"]

As the api has a SearchDocument with oneOf result types, the search returns an array of SearchDocuments and the actual mapped object is the attribute actual_instance.

paginate search could iterate through the search results and return a list of the actual_result items so API users wouldn't have to. This is a decision point. If I were to proceed be good to know which way to go. My initial preference would be to return the actual items rather than force the API user to do so. what is the direction dev rel is going on this?

types had to be added for event actor and target as these were defined as strings and the json result is objects. In order to implement fully the definition of all search results has to be reviewed, possibly defined, and tested. Access to data is 'lazy' and errors don't show up until they are accessed. Is there a plan to validate the yaml for all search results?

working code

Instead of dicts being return, the actual types are. As stated above the SearchDocument has to be examined for what type of document this is.

so it's now event.actor.name instead of event["actor"]["name"]

    with sailpoint.v3.ApiClient(configuration) as v3_client, \
        sailpoint.beta.ApiClient(configuration) as beta_client:

        search = sailpoint.v3.Search(
            indices=['events'], 
            query=sailpoint.v3.Query(query='type:*'),
            sort=['id'])
        
        searchDocuments = Paginator.paginate_search(sailpoint.v3.SearchApi(v3_client), search=search, increment=100, limit=1000)

        print(len(searchDocuments))

        print(f'{"created":<35} {"name":<40} {"actor name":<20} {"target name":<20}')

        for searchDocument in searchDocuments:
            if isinstance(searchDocument.actual_instance, sailpoint.v3.models.EventDocument):
                event = searchDocument.actual_instance
                created = event.created.isoformat()
                name = event.name
                actor_name = event.actor.name or ''
                target_name = event.target.name or ''

                print(f'{created:<35} {name:<40} {actor_name:<20} {target_name:<20}')
            else:
                print(type(searchDocument.actual_instance))

changes made

api yaml

diff --git a/idn/v3/schemas/search/model/event/EventDocument.yaml b/idn/v3/schemas/search/model/event/EventDocument.yaml
index d100e40..f3c99fc 100644
--- a/idn/v3/schemas/search/model/event/EventDocument.yaml
+++ b/idn/v3/schemas/search/model/event/EventDocument.yaml
@@ -26,13 +26,9 @@ allOf:
         description: Event type. Refer to [Event Types](https://documentation.sailpoint.com/saas/help/search/index.html#event-types) for a list of event types and their meanings. 
         example: SYSTEM_CONFIG
       actor:
-        type: string
-        description: Name of the actor that generated the event. 
-        example: System
+        $ref: "./ActorType.yaml"          
       target:
-        type: string
-        description: Name of the target, or recipient, of the event. 
-        example: Carol.Adams
+        $ref: "./TargetType.yaml"
       stack:
         type: string
         description: The event's stack. 

paginate_search.py

as values are attributes and can't be subscripted the paginate_search.py needed this change.

generator command line

Note: --additional-properties=useOneOfDiscriminatorLookup=true

Without this, the generated code tries to map the objects based on attributes. Based on current yaml that results is ambiguous. Unknown if this would be required after evaluating all the search results.

java -jar openapi-generator-cli.jar generate -i api-specs/idn/sailpoint-api.v3.yaml -g python -o . --global-property skipFormModel=false --config sdk-resources/v3-config.yaml --enable-post-process-file --additional-properties=useOneOfDiscriminatorLookup=true  

Next steps, assuming this would be incorporated into the api spec and python sdk would be validate the remainder of the yaml definitions for all search results.

assumptions

1. if an attribute is added to any object, it wouldn't be a breaking change. if an attribute is removed, or the type changes, it would be.
2. internal ISC api calls can handle the search result yaml adjustments
3. other sdks can handle the search yaml adjustment

@tyler-mairose-sp , your thoughts?