As Province of Bolzano I would like to integrate the statistics of usage of the suedtirolRadelt APP in the Open Data Hub, so to create a web-component and visualize them

[rcavaliere]

240524_SpecificheIntegrazione_NOI_v1.1.pdf

[rcavaliere]

@dulvui @clezag we should include this activity in the next sprint. If you have time, have a look at the specification document, on how we should integrate this dataset. I am still clarifying some open points with the Data Provider.

[sseppi]

We decided to externalize the development and @rcavaliere will manage the market research.

[rcavaliere]

@clezag @dulvui can you please support @Giacomo92 and @SanaviaNicolas in the implementation of their first Data Collector? They have some technical questions for you in relation on how certain aspects can be implemented. I would be ideal if you can arrange some short meeting with them in the next days. Thanks!

[dulvui]

@rcavaliere We had a short talk yesterday at the Open Data Hub day and we decided that we will have a meeting with them to clarify some aspects on how to write data to the Open Data Hub.
Can you send me an email with their contacts? So I can arrange a video-call

[Giacomo92]

240207_SpecificheIntegrazione_NOI_v1.pdf
@rcavaliere I have just uploaded the specification document with some comments as we agreed in our last call.

[rcavaliere]

@Giacomo92 here the document with the clarifications, not in track change mode, but hope you find all clarifications to you doubts. Actually, they are all minor points.

240524_SpecificheIntegrazione_NOI_v1.1.pdf

[Giacomo92]

I have just opened PR #669. Please let me know if it works correctly. @dulvui
Here are some questions for @rcavaliere :

• What parameters need to be called on both endpoints? Let me know which data you are interested in retrieving from the web services
• On organisation webservice the field id-challenge_id (280 in the example) is not unique as station id. In my opinion it is better to choose id field (49256209, in the example).

[rcavaliere]

@Giacomo92 many thanks. We will integrate the PR at the beginning of next week, than I will provide you a feedback to these 2 points

[dulvui]

@rcavaliere @Giacomo92 I deployed now the first version of the data collector in testing
Here the first visible GamificationAction stations
https://mobility.api.opendatahub.testingmachine.eu/v2/flat,node/GamificationAction/

And here the CompanyGamificationAction stations, but no data is visible here yet
https://mobility.api.opendatahub.testingmachine.eu/v2/flat,node/CompanyGamificationAction/

[sseppi]

Uau nice to see a new Dataset in the Open Data Hub.

@rcavaliere once we put online the new collector, we need to add the Dataset into the MetaData API to see it in the Dataset List on the Data Browser and the Open Data Hub website. There we need the following information:

• dataset name
• dataset description
• data provider

[rcavaliere]

@Giacomo92 can you please check this implementation? The source should provide data for both type of stations! More in detail:

• In the case of GamificationAction, we have just the station description but no types and data associated to it.
• Moreover we still need to agree on the "position" data provision (via the CSV) file.
• the fields "start", "end", "registrationStart", registrationEnd" in the metadata should be converted in a user-friendly format before stored
• in the case of CompanyGamificationAction we have no stations stored, and of course no types and data associated

[Giacomo92]

@rcavaliere

In the case of GamificationAction, we have just the station description but no types and data associated to it.

Ok I think it depends on the parameters that were passed in the API, if you can tell me which data you are interested in, I will already set the correct parameters on the API call

Moreover we still need to agree on the "position" data provision (via the CSV) file

challenge_id, latitude, logitude for GamificationAction
organisation_id, latitude, logitude for CompanyGamificationAction

the fields "start", "end", "registrationStart", registrationEnd" in the metadata should be converted in a user-friendly format before stored

Which format do you prefer? I chose the timestamp by looking at other implementations in the repository

in the case of CompanyGamificationAction we have no stations stored, and of course no types and data associated

Ok I think it depends on the parameters that were passed in the API, if you can tell me which data you are interested in, I will already set the correct parameters on the API call

[rcavaliere]

Ok I think it depends on the parameters that were passed in the API, if you can tell me which data you are interested in, I will already set the correct parameters on the API call

As already described in the specification document, these are the types to be associated to a station = GamificationAction:

challenge_id, latitude, logitude for GamificationAction
organisation_id, latitude, logitude for CompanyGamificationAction

OK, but where should I put this information? Can you create a CSV file (or similar), so that I fill in the positions?

Which format do you prefer? I chose the timestamp by looking at other implementations in the repository

Please use this kind of format, as you can see there for the dateTime fields: https://mobility.api.opendatahub.com/v2/flat,node/BikesharingStation/*/latest

Ok I think it depends on the parameters that were passed in the API, if you can tell me which data you are interested in, I will already set the correct parameters on the API call

Please have a look at the specification document, consider what provided in the paragraph COMPANY GAMIFICATION ACTIONS

I put the specification document again here. Please let me know in case of doubts, in case let us have a quick call

240524_SpecificheIntegrazione_NOI_v1.1.pdf

[dulvui]

@rcavaliere @Giacomo92 Also the measurements are now visible here

CompanyGamificationAction
https://mobility.api.opendatahub.testingmachine.eu/v2/flat,node/CompanyGamificationAction/*/latest?where=sactive.eq.true&limit=-1

GamificationAction
https://mobility.api.opendatahub.testingmachine.eu/v2/flat,node/GamificationAction/*/latest?where=sactive.eq.true&limit=-1

[rcavaliere]

@dulvui thanks for the update. I have checked the data flowing in our testing environment, and I have seen following aspects that need to be fixed:

• the fields End, Start, registrationEnd, registrationEnd (metadata of GamificationAction station): are supposed to be timestamps, so we need to format them properly so to let them immediately comprehensible and usable
• the field name of stations CompanyGamificationAction is mapped in a wrong way, has always the same value for all stations. The idea is to concatenate here the name of the organisation and the name of GamificationAction (fields name and challenge_name), we should have something like "Comune di xxx - Alto Adige Pedala 2024".

@dulvui let me know if this something you can easily fix, otherwise we will ask @Giacomo92 to have a look at it.

[dulvui]

@rcavaliere I fixed now both issues

I also saw that now that there are two urls we use currently
https://www.suedtirolradelt.bz.it/dashboard/api/opendata/challenges
https://www.altoadigepedala.bz.it/dashboard/api/opendata/challenges

One url gives the results in German and the other in Italian. The German is currently used to get companies data and the Italian on for the challenges data. Is this okay and no problem if there are multiple languages or should I use only Italian or German?
Using both for both calls would also be possible, but for that some additional work is needed for mapping and the data structure has to be defined on how both languages will be mapped.

[rcavaliere]

@dulvui thanks for the feedback. For the multilingual aspect, thanks for the input. If feasible, I would consider this just for the names of the stations. Can you implement this?

[dulvui]

@rcavaliere Okay I'll try to get use both languages for the names, and for the rest I'll only use one.
Should not be too complicated, because I just need to make one more call with the other language and merge the 2 names to one, for the rest I'll not change anything.

[rcavaliere]

@dulvui thanks. However consider for the two type of stations the same language! Let's use German....

[rcavaliere]

@dulvui @Giacomo92 another thing: can you put me the link to the file for the provision of the coordinates of the stations?

[Giacomo92]

Hello, thanks @dulvui for the fixes. @rcavaliere this is the links:

• organizations
• actions

[dulvui]

@rcavaliere station names are now in German and Italian

[rcavaliere]

@dulvui perfect, now the open points are closed

[rcavaliere]

@dulvui please don't put this Data Collector in production, we need first to sign the Data Sharing contract with STA, owner of Suedtirol Radelt APP

[rcavaliere]

@dulvui @clezag can you check if the Data Collector is consuming the right points? We don't get data since 12.8, and the measurement associated to CompanyGamificationAction is nearly not available. Can you please check? I think the problem could be just in the reference to the right end-points?

The correct ones (please check the specs in the user story description):
https://www.suedtirolradelt.bz.it/dashboard/api/opendata/challenges
https://www.suedtirolradelt.bz.it/dashboard/api/opendata/organisations

[clezag]

@rcavaliere The data collector was permanently stuck doing the API request. Seems that the library used does not use a timeout per default. I've set a timeout of 30 seconds, so let's see if this crops up again. For now, since it's been restarted, it's collecting data again

[rcavaliere]

@clezag thanks for this. If I look at the measurementhistory table it seems to me that we just imported one record for each type, and then the Data Collector did not import any other data. Is there any logic that prevents the storage of new data every time the API is read? We could set here a hourly update of the data, we don't need this more frequently

[clezag]

@rcavaliere
The issues are twofold I think:

• we're using the created_at field as our measurement timestamp, which means that as long as that field does not change, a new record is not written
• we're using the statistics.challenge.id as station name, which I think changes everytime the statistics are updated. We should probably use just the challenge ID as identifier.
  As can be seen here: https://mobility.api.opendatahub.testingmachine.eu/v2/tree/CompanyGamificationAction/number_of_people/2024-08-07/2024-09-18?referer=test&limit=-1&where=sname.eq.%22Gruppo%20Sportivo%20Fortezza-S%C3%BCdtirol%20radelt%202024%22 filtering for the challenge name shows that the same challenge is recreated as station every time there is an update, so naturally, considering the issue with timestamp, every station always has exactly one record per data type

I think to avoid data duplication, we can leave 1) as it is, so that only when something changes, a measurement is written

[rcavaliere]

@clezag then please clean up all the DB in the testing environment, use as key the challenge ID and try to redeploy it. Thanks!

[clezag]

@rcavaliere Now up in testing. I've removed the old data.
The stationcode is now organization_id + challenge_id (I mistakenly wrote only challenge_id above, which is not unique)

[rcavaliere]

@clezag thanks a lot. In measurementhistory I currently see just one value per type in measurementhistory, but it could be that nobody used the Suedtirol Radelt APP yesterday afternoon and registered new kilometers... let's see in the next days if new data is entering, otherwise it could be that we have still things to fix.

[rcavaliere]

@clezag everything tested here and contrat signed, we can put in production

[clezag]

@rcavaliere deployed in production

[sseppi]

@rcavaliere can you please share with us the information to create the dataset on our website?

We need:

• API call with the right filter to get the data
• Dataset Name
• DAtaset Description

Thanks