Questions regarding data transfer in the Dataspace Connector

[Vincent-UBT]

Hi everyone,

while testing the latest version of the Dataspace Connector (b75efa9), I encountered three problems which I would like to discuss here.

1. I did not manage to register an 'http-get' resource. I tried amending the provided JSON with the 'local' example by exchanging 'local' through 'http-get' and adding the fields 'url', 'username', and 'password'. I The url I used was taken from an earlier example in the Dataspace Connector (https://samples.openweathermap.org/data/2.5/weather?lat=35&lon=139&appid=439d4b804bc8187953eb36d2a8c26a02).

Does anyone know where my mistake is or whether this is a bug in the current release?

2. Is 'PUT /admin/api/resources/{resource-id}/data' actually the intended way to upload data to an internal database?

2.1 Having to upload data as a String might potentially be a limitation or lead to overhead. Does anyone agree or am I missing something here?
2.2 I get errors when trying to upload very long Strings.

3. The iterative protocol required when trying to obtain an artifact from another connector seems quite heavy. I encountered issues when trying to automate the workflow, as parsing the necessary data in the responses obtained from 'POST /admin/api/request/description' and 'POST /admin/api/request/artifact' is quite difficult.

Potentially this protocol could be improved or the data in the responses structured in a more forward way?

[brianjahnke]

Hello,

we (@ronjaquensel and me) have been capable of registering a resource successfully.

{
  "title": "ExampleResource",
  "description": "ExampleResourceDescription",
  "policy": "Example policy",
  "representations": [
    {
      "type": "XML",
      "byteSize": 101,
      "name": "Example Representation",
      "source": {
        "type": "http-get",
        "username": "",
        "password": "",
        "url": "https://samples.openweathermap.org/data/2.5/weather?lat=35&lon=139&appid=439d4b804bc8187953eb36d2a8c26a02"
      }

    }
  ]
}

[Vincent-UBT]

Hello Brian,

this example worked for us as well. We might have made a syntax error by writing "-" as value for 'username' and 'password'. Thank you very much!

Do you have any thoughts on the remaining comments?

Best Regards
Vincent

[brianjahnke]

Hi Vicent,

we are aware that the current data model has limitations and it's part of our roadmap. In fact we are currently working on the endpoints.

In regards to point 2.2 "very large" please be more precise but regardless of the specific number a limit for the string length should always be expected.

As for point 3. I cannot judge how hard it is, since i have limited insight into setting up an automatic workflow. But i know of other projects that have done so successfully. Have you checked out the infomodel serializer mentioned in our readme?
If you have suggestions on how improvements should look like feel free to share them with us (we always appreciate constructive input!) or contribute yourself!

-Brian