Fix On-Street Parking Data Collector

[rcavaliere]

No description provided.

[dulvui]

I tested now the push endpoint and it works fine. I send the test data that is available in the repository and it worked.
The endpoint I used is https://parking.opendatahub.bz.it/dc-onstreet-parking-bz/json/pushRecords

Could it be that there are problems on data provider site?

[rcavaliere]

@dulvui yes it could be, this is the feedback I wanted to get. I will contact the Data Provider and let you know.

[rcavaliere]

News here: the tech provider here (Axians) is changing platform and will push data back soon. Probably we will need to change something (little things) in our Data Collector

[ohnewein]

Waiting for Axions to provide the new API which we have to use.

[rcavaliere]

@clezag we have some news here. The new end-point is mqtts://xsona-broker.axians.it:8443, we must subscription to the topic application /6f599389-c6b0-4a76-843d-8d493192c4c0/device/+/event/up. I provide you the credentials separately. Please note that the sensors are still not connected to this back-end, we should now just test if the connection is working

[clezag]

@rcavaliere this is the "parking-onstreet-bz" data collector (origin=AXIANS), right?
As far as I understand, it currently exposes a REST endpoint that they used to push data, but now we need to connect to a MQTT broker instead?
Just making sure I'm not looking at the wrong thing

[rcavaliere]

@clezag yes, correct. No, it will work in the past, with them pushing us the data. Just check now if the connection is OK.

[clezag]

@rcavaliere I've tried connecting with two tools, but I get timeouts:

If you have docker installed you can try, too:

docker run hivemq/mqtt-cli subscribe \
--topic '/6f599389-c6b0-4a76-843d-8d493192c4c0/device/+/event/up' \
--host xsona-broker.axians.it --port 8443 \
--identifier 'noitest' \
--user '***' \
--password '****' \
--secure -v

(it doesn't even go as far as checking the credentials)

Maybe they can fix this command if something is missing and send it back to us?

[rcavaliere]

@clezag thanks for the check, I have notified the issue to the Data Provider. Let's put the task in stand-by until we have a feedback.

[rcavaliere]

@clezag the technological provider of the platform asks to have a troubleshooting session in order to more quickly solve the issue. Can you manage this? If yes, I will put you in contact with the reference software developer.

[clezag]

@rcavaliere sure, just send me their contact

[rcavaliere]

@clezag just shared via e-mail the reference contacts of Axians

[clezag]

@rcavaliere We've just had a call and end the end we were just using the wrong port. Correct is 8883 and not 8443.

Now I'll wait and check if we get some data and how it looks

[clezag]

We're now receiving data from the MQTT, still have to implement/migrate the transformer part.
For completeness, these are the parameters that work (topic and port were wrong before):

docker run hivemq/mqtt-cli subscribe \
--topic 'application/6f599389-c6b0-4a76-843d-8d493192c4c0/device/+/event/up' \
--host xsona-broker.axians.it --port 8883 \
--identifier 'noitest' \
--user '***' \
--password '******' \
--secure

[clezag]

This is an example JSON:

{
  "deduplicationId": "4cb994bf-5141-4b21-ac09-4cda8a1fd628",
  "time": "2024-11-14T07:34:17.899236+00:00",
  "deviceInfo": {
    "tenantId": "b74ae308-29c2-40c2-8036-46e422b7da4d",
    "tenantName": "Comune di Bolzano",
    "applicationId": "6f599389-c6b0-4a76-843d-8d493192c4c0",
    "applicationName": "parking",
    "deviceProfileId": "d3c3cb95-d41d-4af1-8ec0-56fe8005d24c",
    "deviceProfileName": "Intercom-RSPS-WSM-SREVH-R56",
    "deviceName": "815",
    "devEui": "001bc50670100519",
    "deviceClassEnabled": "CLASS_A",
    "tags": {}
  },
  "devAddr": "00b67edd",
  "adr": true,
  "dr": 3,
  "fCnt": 28,
  "fPort": 1,
  "confirmed": true,
  "data": "gAAAAy8S9AABZnAacQE6AAA6jAAi3G8A4lc=",
  "rxInfo": [
    {
      "gatewayId": "00800000a0005ca9",
      "uplinkId": 2777845975,
      "gwTime": "2024-11-14T07:34:17.899236+00:00",
      "nsTime": "2024-11-14T07:34:17.911275802+00:00",
      "timeSinceGpsEpoch": "1415604875.899236s",
      "rssi": -113,
      "snr": -7.5,
      "location": {
        "latitude": 46.49902,
        "longitude": 11.35597,
        "altitude": 294.0
      },
      "context": "AAAAAAAAAAAAAgAc7BRtVA==",
      "metadata": {
        "region_common_name": "EU868",
        "region_config_id": "eu868"
      },
      "crcStatus": "CRC_OK"
    }
  ],
  "txInfo": {
    "frequency": 868100000,
    "modulation": {
      "lora": {
        "bandwidth": 125000,
        "spreadingFactor": 9,
        "codeRate": "CR_4_5"
      }
    }
  }
}

In the past, it looked more like this:

...
            "data": "gAAAAywS9AABXlPUYgE6AAADJAAANfcA1t8=",
            "parsedData": {
                "Magnitude": 804,
                "Payload": "MEASURE",
                "Progtx": 13815,
                "Retries": 0,
                "Sensor-address": "0000032c",
               "Status": "free",
                "Time": 1582552162,
                "Version": 58
            },
...

Crucially, the parsedData.Status field/structure is nowhere to be found, and the data field is in binary, so I don't know how to interpret the actual sensor data.
I have contacted the data provider requesting documentation on how to decode it.

[clezag]

The data provider gave us the documentation, which should suffice to implement the data collector.

@rcavaliere I've noticed that this uses a google spreadsheet for station metadata, but it hasn't been updated since 2020.
There is also all kind of generic mapping code for metadata and whatnot, but looking at the station, this isn't used.
The municipality is retrieved using the nominatim webservice.

Would it be OK if I move this to a static CSV/yaml file within the repo to keep things simple and make the migration a lot faster? And just add the municipality as another field (it's always Bolzano anyway).

[clezag]

@rcavaliere I've implemented the transformer using a simple CSV file (copied from the old spreadsheet).
it's already running and some data is visible in testing (to me it looks like only a few sensors are active, but I have not verified that this corresponds to the data)

[rcavaliere]

@clezag many thanks. Let me analyze your work in detail before making further steps here