How to deal with dynamic/complex REST services

[hylkevds]

Hi All,
This issue is mostly to document our discussion.
Currently, when using the DataspaceConnector on the server side, to expose an existing REST service, one has to specify the exact REST request that corresponds to a given Resource. However, many REST APIs require the passing of query parameters to be useful.

For example the service at https://airquality-frost.k8s.ilt-dmz.iosb.fraunhofer.de/v1.1 provides air quality measurements for all of Europe. A typical use-case for this service would be:

1. Request Things from https://airquality-frost.k8s.ilt-dmz.iosb.fraunhofer.de/v1.1/Things
   Either all, or using a filter.
   • Stations that measure NO2: https://airquality-frost.k8s.ilt-dmz.iosb.fraunhofer.de/v1.1/Things?$filter=Datastreams/ObservedProperty/Name eq 'NO2'
   • Stations in a certain bounding box: https://airquality-frost.k8s.ilt-dmz.iosb.fraunhofer.de/v1.1/Things?$filter=geo.intersects(Locations/location,geography'POLYGON ((0 55.7,0 52.4,5.6 52.4,5.6 55.7,0 55.7))')
2. Fetch, when needed, for a station the Datastreams, with ObservedProperty
   https://airquality-frost.k8s.ilt-dmz.iosb.fraunhofer.de/v1.1/Things(2281)/Datastreams?$select=name,id,unitOfMeasurement,observationType,properties&$expand=ObservedProperty
   The 2281 would be replaced for each call.
3. Fetch the Observations from a Datastream, for a given time-frame.
   https://airquality-frost.k8s.ilt-dmz.iosb.fraunhofer.de/v1.1/Datastreams(245)/Observations?$filter=phenomenonTime ge 2020-08-01T00:00:00Z and phenomenonTime lt 2020-09-01T20:00:00Z&$select=result,phenomenonTime&$orderby=phenomenonTime asc
   Again, the 245 would be replaced for each call, the times depend on the use case.

In all of these requests, other parameters can be used to tune the data that is returned, both the minimise the number of requests required, and to minimise the amount of data that needs to be sent. For instance, with the $select and $expand parameters.

Furthermore, for each request that returns a List of items, the response can be a subset of the list, and contain a @iot.nextLink property, that points to the next subset. This nextLink is dynamically generated by the server. The DataspaceConnector on the client side would have to replace those links with links pointing to itself.

The service contains data from January 2018 till now, about 300 Million Observations, in 17974 Datastreams, for 4382 Stations.
A demo map with the data of this service can be found at https://datacoveeu.github.io/API4INSPIRE/maps/AirQuality.html
The complete API standard document can be found here: https://www.ogc.org/standards/sensorthings