Tasks API

This section describes the task API for communication with IVIS core. IVIS currently supports only Python for programming tasks (that will likely change in the future); therefore examples will be shown only for this case.

WARNING: If you are just interested in using the tasks you don't have to study the API details deeply. We have a Python package that handles it for you! Details below.

Task code

For our code to be useful in the IVIS framework we need a way to communicate with IVIS core. All the communication is set up in a task's code. There will be explanation for the underlying technical details but you won't use them during actual development as you will be using the IVIS Python package. This package significantly simplifies all communication with IVIS core.

API details

Primary attributes

Each job once started receives attributes in JSON format on the standard input, file descriptor 0. Here is a simple code for task to see what the input data are:

import sys
import json

data = json.loads(sys.stdin.readline())
print(json.dumps(data, indent=4, sort_keys=True))

Example of data:

{
    "accessToken": "xxxxxx",
    "context": {
        "jobId": 1
    },
    "entities": {
        "signalSets": {
            "created_set_1": {
                "cid": "created_set_1",
                "created": "2021-10-24T18:48:24.000Z",
                "description": "Example created set",
                "id": 2,
                "index": "signal_set_2",
                "kind": "generic",
                "metadata": null,
                "name": "created_set_1",
                "namespace": 1,
                "record_id_template": null,
                "settings": "{}",
                "state": "{\"indexing\":{\"status\":0}}",
                "type": "computed"
            },
            "set1": {
                "cid": "set1",
                "created": "2020-06-15T08:57:33.000Z",
                "description": null,
                "id": 1,
                "index": "signal_set_1",
                "kind": "time_series",
                "metadata": null,
                "name": "set1",
                "namespace": 1,
                "record_id_template": "{{toISOString ts}}",
                "settings": "{\"ts\":\"ts\"}",
                "state": "{\"indexing\":{\"status\":0}}",
                "type": "normal"
            }
        },
        "signals": {
            "created_set_1": {
                "created_signal": {
                    "cid": "created_signal",
                    "created": "2021-10-24T18:48:25.000Z",
                    "description": "created_signal",
                    "field": "s2",
                    "id": 2,
                    "indexed": 0,
                    "name": "created_signal",
                    "namespace": 1,
                    "set": 2,
                    "settings": "{}",
                    "source": "job",
                    "type": "double",
                    "weight_edit": null,
                    "weight_list": null
                }, 
            }
        }
    },
    "es": {
        "host": "localhost",
        "port": "9200"
    },
    "owned": {
        "signalSets": {
            "created_set_1": {}
        }
    },
    "params": {
        "sigSet": "set1"
    },
    "server": {
        "sandboxUrlBase": "http://localhost:8444",
        "trustedUrlBase": "http://localhost:8443"
    },
    "state": {
        "test": "test"
    }
}

There are 8 main JSON objects in the incoming data:

• accessToken
• context
• entities
• es
• owned
• params
• server
• state

accessToken is token that can be used to access REST API of the core system.

context contains information about current run.

entities have 2 JSON objects, signalSets and signals. Each signal set found in the parameters of the job (or being owned) is listed in the signalSets under his CID with its properties (index is corresponding index in the ElasticSearch). Each signal found in the parameters (or being owned) has its properties listed in the signals object with path signal_set_cid->signal_cid (field is field in ElasticSearch in the index of its signal set).

es contains information about ElasticSearch instance. Under host is host's address and under port is port it is listening on.

owned contains all the entities created by this job. This process is described in the requests section.

params are parameters from the job being run that were set previously in GUI of the job's settings page. It is always a pair of parameter's identifier and selected value.

server contains information about IVIS instance like its URLs.

state is job's state stored in the ElasticSearch. Content of state depends completely on what is stored there in the job's code. More on that later.

Job requests

Job can send request to the IVIS core. Requests are accepted on the file descriptor 3 in JSON format and answer is received on the standard input. There are 2 types of requests job can send:

• store_state
• create_signals

store_state will request storing given state. This state is received on each run start in state object mentioned previously. create_signals will request creating new signal set and signals. Example of a store_state request:

{
 "id": 1,
 "type": "store_state",
 "state": {
   "index": "signal_set_2",
   "type": "_doc",
   "fields": {
     "created_signal": "s3"
   }
 }
}

and received answer:

{
  "id": 1
}

Each request has type, either store_state or create_signals, that determines requested action. If there is id present in the request it will be copied to the answer unless the JSON format was incorrect. If there is no error present, request succeeded. Otherwise error with error message will be present in the answer:

{
  "id": 1,
  "error": "Request failed"
}

In the store_state request everything in state object will be stored.

create_signals request looks like this:

{
  "type": "create_signals",
  "signalSets": {
      "cid" : "created_set",
      "name" : "Example set" ,
      "namespace": 1,
      "description" : "Documentation example signal set" ,
      "signals": [
        {
          "cid": "created_signal",
          "name": "Example signal",
          "description": "Documentation example signal set",
          "namespace": 1,
          "type": "text",
          "indexed": false,
          "settings": {} 
        }       
      ]
  }
}

and received answer:

{
  "created_set": {
    "index": "signal_set_2",
    "type": "_doc",
    "signals": {
      "created_signal": {
        "field": "s3"
      }
    }
  }
}

It is possible to request multiple signal sets at once:

{
  "type": "create_signals",
  "signalSets": [
    {
      "cid" : "created_set1",
      "namespace": 1,
      "signals": [
      ]
    },
    {
      "cid" : "created_set2",
      "namespace": 1,
      "signals": [
      ]
    }
  ]
}

answer:

{
  "created_set1": {
    "index": "signal_set_23",
    "type": "_doc",
    "signals": {
     }
  },
  "created_set2": {
    "index": "signal_set_24",
    "type": "_doc",
    "signals": {
     }
  }
}

It is also possible to request creation of signals in an existing signal set:

{
  "type": "create_signals",
  "signals": {
    "existing_signal_set_cid": [
      {
        "cid": "created_signal",
        "name": "Example signal",
        "description": "Documentation example signal set",
        "namespace": 1,
        "type": "text",
        "indexed": false,
        "settings": {} 
      }   
    ]   
  }
}

answer:

{
  "existing_signal_set_cid": {
    "index": "signal_set_16",
    "type": "_doc",
    "signals": {
      "created_signal": {"field": "s25"}
     }
  }
}

Once the requested signal sets or signals are created they appear in the owned attribute:

"owned": {
  "signalSets": {
    "created_set1": {}
  }
}

IVIS Python package

Instead of using the described API directly we will utilize IVIS Python package. This library simplifies communication with the IVIS core. First we have to import it:

from ivis import ivis

Previously mentioned attributes are available as properties on ivis object. The es attribute is elasticsearch property and is instantiated Elasticsearch class from the elasticsearch package.

es = ivis.elasticsearch
state = ivis.state
params = ivis.params
entities = ivis.entities
owned = ivis.owned

Example code accessing property:

import json
from ivis import ivis

print(json.dumps(ivis.entities["signalSets"]["process1"], indent=4, sort_keys=True))

would give you:

"set1": {
   "cid": "set1",
  "created": "2020-06-15T08:57:33.000Z",
  "description": null,
  "id": 1,
  "index": "signal_set_1",
  "kind": "time_series",
  "metadata": null,
  "name": "set1",
  "namespace": 1,
  "record_id_template": "{{toISOString ts}}",
  "settings": "{\"ts\":\"ts\"}",
  "state": "{\"indexing\":{\"status\":0}}",
  "type": "normal"
}

Similarly you can access others.

Task requests

There are several helper functions for task's requests. For creating signal sets and signals there are:

ivis.create_signal_set(cid, namespace, name=None, description=None, record_id_template=None, signals=None)
ivis.create_signal(signal_set_cid, cid, namespace, type, name=None, description=None, indexed=None, settings=None, weight_list=None, eight_edit=None, **extra_keys)

These will create signal set and signal with given properties respectively. There is also functions for creating multiple at the same time.

ivis.create_signals(signal_sets=None, signals=None)

At least one of signal_sets or signals have to be set. On successful creation owned and entities attributes are updated with received specs of the newly created entities.

There is also function for storing state. State can be dictionary with whatever data you need.

ivis.store_state(state)

You can also work with files. upload_file will store and associate given file with current job. get_job_file will request file with given id.

ivis.upload_file(file):
ivis.get_job_file(id):

Submodule for neural networks

The package has a submodule ivis.nn which can be used to create prediction models based on neural networks. It is used by the Prediction models available in IVIS. Details can be found in the README.md file in the directory with the package code.