As MaaS Operator, I would like to have a secure end-point from which I can retrieve anonymous user codes

[rcavaliere]

In short time MaaS Operators will be able to offer MaaS services to local users thanks mainly to the Open Data Hub.
The project provides incentives to users testing such APPs. For being part of the project, selected users of selected companies will have to register thanks to a Microsoft Form managed by STA. Thanks to the answers obtained, users' identifiers can be generated - these are necessary for the interface between the MaaS operator back-end and the national MaaS platform, where for statistical / research reasons the users' trip patterns are stored. Everything is GDPR-compliant! The user inserts a user code which is securely provided by the mobility manager of the company.

The MaaS Operator would need to have instead of a CSV / Excel file an API with just the relevant data needed. So the user story would be:

• read the CSV from an online link: https://stabz.sharepoint.com/:x:/r/InfoSys/ProjekteIS/P005200/InternalDocuments/240920%20Forms%20-%20raccolta%20dati.xlsx?d=w089c6cfa5a0742779a6abefbb90d714b&csf=1&web=1&e=ovdT4v
• design the API by considering just the relevant fields (to be defined)
• implement and publish the API on an end-point

No data storage is needed in the Open Data Hub; in this case we can directly publish this data.

Possibly within October 4th.

[rcavaliere]

@clezag @luhodaan the API should simply retrieve as JSON a list of objects, each object should contain three fields:

• codiceUtente
• idViaggiatore
• tipoViaggiatore

codiceUtente is a field given in the Excel file linked above, while idViaggiatore and tipoViaggiatore are something to be calculated based on the answers given by a user to the form, and available in the Excel file. I will provide you tomorrow the exact way we have to calculate these two codes, with an example. In the meantime you can start trying to read the content of the Excel file (see link above in the user story description)

[rcavaliere]

@clezag @luhodaan now if you see online link below you can find the exact way these codes (idViaggiatore and tipoViaggiatore) have to be computed. You can find some example so that you can understand better. In the attached specification you can find the exact rules for the generation of the code tipoViaggiatore.

DSSRF-Gestione viaggi_v2.2_signed-17-19.pdf

[luhodaan]

@rcavaliere From the STA CSV file I am using locally to test the api, I see already present the 2 fields that should be instead computed as understood from the above comments.

I refer to:

• idViaggiatore
• tipoViaggiatore
  Any idea on this point?

Also as soon as anything comes out about the authentication methods for the STA sharepoint, I'll move from the local csv to the actual data.

[rcavaliere]

@luhodaan good, but I would ask you to compute these two codes as implemented in the Excel file, by considering the formulas that you can see implemented there. I can provide you some additional explanations by voice if needed

[luhodaan]

@rcavaliere the api is now in production. At the moment it is taking the data from the csv copy you sent me. So the only thing left should be getting the data from source instead when the sharepoint wil be accessible.

[rcavaliere]

@luhodaan. Wonderful. Can you please share me the link here?

[luhodaan]

@rcavaliere https://maas4italy-users.api.opendatahub.testingmachine.eu/users

there is also an endpoint to filter results from the known Codice utente:
https://maas4italy-users.api.opendatahub.testingmachine.eu/users/:CodiceUtente

[rcavaliere]

@luhodaan in the repository there is a folder in which you actually read the CSV file, right? Can you put me the link here (of the repository and where the file is placed)?

[luhodaan]

@rcavaliere https://github.com/noi-techpark/maas4italy-pilot-user-api/tree/main.
The source csv file is not directly accessible to API users and it's not in the repo.

[rcavaliere]

@luhodaan it would be good to have the CSV file in the repo. Maybe let's make the repo closed and not of public domain, so that we only as team can have access to the file

[rcavaliere]

@luhodaan I have spoken with STA about the access to the Excel file. There is no way to do differently, you should access via this link: https://stabz.sharepoint.com/:x:/g/InfoSys/ProjekteIS/P005200/EfpsnAgHWndCmmq--7kNcUsBhCWUw3FRvdxpYdM9khmlLw?e=eQkJfl

Your email address (and also of @clezag) are now enabled to access the file.

In the meantime, in order to not block the activities because of this automatic interface with the online Excel, I would recommend to the following:

• the application reads the CSV from a certain folder in the repository
• every time the CSV is uploaded with new records, the end-point is updated with the new codes

So, it is not fully automatic but e.g. we can update the file once a day.

Can you please complete this by the end of the week please? In the meantime I put here an updated version of the CSV, can you let the end-point be available with all these records please?

240920 Forms - raccolta dati(Tabelle1).csv

[luhodaan]

@rcavaliere I have updated this morning the logic to fetch the data. Now each time that a new data.csv file is uploaded in the repo (the old one needs to be discarded or renamed), the ci/cd takes care of pulling the new relevant data.
it is important that the csv filename keeps consistent and with the name of data.csv
I've deployed it also in production at the following url:
https://maas4italy-users.api.opendatahub.com/users
So in the meantime we can already proceed with this way of substituting the file once a day.

I'll further look how to move in the direction of substituting this mechanism with the online Excel access.

[rcavaliere]

@luhodaan perfect, thanks! Where should we put the CSV exactly?
Another thing: I have seen that the code idViaggiatore is currently read from the file, but I think there is something wrong. I would also automate this by reading the field in the column Azienda . At the end is formatted simply as: ITH10_ValueFieldAzienda where ValueFieldAzienda is what you read in the coloumn Azienda.

[luhodaan]

@rcavaliere the CSV has to be put in the outer level of the repo, just like the README or the gitignore, you can also see it in the current organisation of the repo: https://github.com/noi-techpark/maas4italy-pilot-user-api/tree/prod.

You're right, i fixed both idViaggiatore and tipoViaggiatore, which weren't consistent in how they were computed.
Now they should be fine.

[rcavaliere]

@luhodaan I have tried to update the CSV file since we are starting to have now many users registering to the pilot project. However I can not see an update in the API, can you check? I have also seen that the last two codes associated to Markas are not correctly generated (see records #14 and #15). One last thing: in one of the new answers a person has selected as answer for question AA the reply '2' (which stands for "I don't want to reply"). Please ensure in this case that last digit in this case is zero, since we have always to guarantee that the code TipoViaggiatore has exactly 25 digits.

[luhodaan]

@rcavaliere The file has to be updated on branch prod to see the updates in production. If it is updated on main, which is for test, the change is visible here: https://maas4italy-users.api.opendatahub.testingmachine.eu/users
I've merged the prod branch to reflect the newly updated data.

I'll take care of the padding for codes shorter than 25 digits, adding zeros to match the length.
Also, I've seen that the last three records of IdViaggiatore don't have the field ITH10_ as the other on the excel, should that be fixed also?

[rcavaliere]

@luhodaan thanks! Yes please also fix the ITH10_ aspect!

[luhodaan]

@rcavaliere now it is fixed. I also noticed a problem with how the csv read the accented characters in the header, which was a problem for fetching correctly the data, but now it should be good.

[rcavaliere]

@luhodaan I have a new request for you. We need to integrate a second CSV file (attached here), which is equal to the other one. The form with which we collect this data from the users needed to be different since in this case the users have to enter slightly different values. Can you please consider also this entry file and make sure that our end-point also provides the codes associated to the users registered also in this way?

data2.csv

[luhodaan]

@rcavaliere I've added the logic to fetch the data from both files. Now the api will show the users from both.
It is important as before that this second file is always uploaded with the consistent name of data2.csv, and that the first one still remains data.csv.

[luhodaan]

@rcavaliere I understand that this method that involves updating the CSV on the repo can be from time to time inconvenient. If you think the API will be alive for a prolonged period of time, we can consider having it fetch data directly from SharePoint somehow. I've been told there's a consultant who's quite experienced with SharePoint and Microsoft products authentication in general, I could try and reach out to him if needed.

[rcavaliere]

@luhodaan maybe it could be interesting to analyze this feature more in detail, so try to involve him! Anyway, I think that with the manual upload of the codes could also work, but please check this possibility, so that we can fully automate the workflow