Premium Providers: Import and Export CSV files

General work flow

Developers

Prerequisites:

• Language specific characters (e.g. kanji) will not be supported and imported into the DB (agreed with Holger, 2017-05-18)
• Strings with 4byte characters, e.g. Emojis, will be truncated before mysql db insert actions

Development ENV:

1. Check the validity of each CSV and modify the columns accordingly
2. Make sure to be in the root of the project directory before starting the rake task
3. For Streetspotr use the streetspotr.rake task with bundle exec rake streetspotr:check file="path-to-filename.csv" to do a dry-run or bundle exec rake streetspotr:import file="path-to-filename.csv" to import data

Staging/Production ENV:

4. SSH into the staging/production server
5. Go to the project root of the current release on staging/production
6. Test the import with a the dry-run via RAILS_ENV=staging/production bundle exec rake streetspotr:check file="path-to-filename.csv"
7. Execute the import with RAILS_ENV=staging/production bundle exec rake streetspotr:import file="path-to-filename.csv"
8. Localize the provided pois via RAILS_ENV=staging/production bundle exec rake poi:locate

9. Execute the rake task RAILS_ENV=staging bundle exec rake premium:upload PROVIDER=<file_name> (This task needs to be done manually and is to display the map bubble 'streetspotr says: This place is fully wheelchair accessible.') on each provided poi (node) of a premium provider. TASK DEPRECATED! Premium providers map bubble will be moved to accessibility.cloud in the future.

IMPORTANT: Since there is a Carrierwave background job running on production each imported photo has the value true while running the image_processing and false when the job is done.

1. Streetspotr

0. Discuss and organize with Streetspotr the requirements (e.g. columns, formats). (Sozialhelden)

   For the CSV import via streetspotr.rake the CSV should contain:

     # Following columns are mandatory for the import rake task:

     - row[:osm_id] # (attr.Id) # this column is necessary to have a unique identifier
     - row[:photo_url] # => this column is necessary to fetch from external streetspotr url and assign it to `.photo_url`
         - should be protocol `https`

     # Additional criteria:

       - data-type: string # => text longer than 255 varchar will be truncated
       - emojis (4byte chars) will be removed before import into the mysql2 database (to prevent mysql2 insert value encoding error)

     # EXIF data
       
       - csv must contain EXIF date_time that is supported by the DateTime parser, for more infos please follow this thread: 'ArgumentError: argument out of range' – https://github.com/sozialhelden/wheelmap/issues/633

1. Create a Github ticket for the csv export according to the requirements. (Sozialhelden)
2. Export wheelmap data to a csv file via rake task and send file to Sozialhelden. (Developers)
3. Seed wheelmap data into the Streetspotr app. (Streetspotr)
4. After Streetspotr-campaign has ended send back final csv file to Sozialhelden. (Streetspotr)
5. Send final csv file to developers. (Sozialhelden)
6. Check validity of final csv file. (Developers)
7. Import csv data into the wheelmap database via rake task. (Developers)
8. Import Streetspotr photos (provided through image-url) into the wheelmap database & localize them via rake task. (Developers)

Useful Links:

• OSM Import guidelines
• OSM Forum to discuss imports