S3 data cleanup

[leothomas]

PR #46 modified the locations of certain data files in order to maintain a 1-to-1 mapping between S3 folders and datasets (ie: each dataset should map to only one folder and each folder should contain only one dataset). This is to enable the dynamic domain extraction to easily query the keys of a given dataset (without having to filter out keys belonging to a different dataset in the same bucket).

These modifications include:

• s3://covid-eo-data/detections/plane --> s3://covid-eo-data/detections-plane
• s3://covid-eo-data/detections/ship --> s3://covid-eo-data/detections-ship
• s3://covid-eo-data/xco2/*mean* --> s3://covid-eo-data/xco2-mean
• s3://covid-eo-data/xco2/*diff* --> s3://covid-eo-data/xco2-diff
• All files with full text spotlight labels ("Beijing" , "NewYork", etc) in s3://covid-eo-data/BM_500M_DAILY have been copied to files using spotlight identifiers ("be", "ny", etc) (All files with label EUPorts were omitted from this operation because the EUPorts label maps to two spotlights (du and gh))
• s3://covid-eo-data/agriculture/CropMonitor* --> s3://covid-eo-data/agriculture-cropmonitor

TODO:

Once the /datasets endpoint has been validated and any other code modifications have been made to ensure that these location modifications do not break anything, the following steps should be taken to avoid data duplication in S3

• delete s3://covid-eo-data/detections/plane
• delete s3://covid-eo-data/detections/ship
• delete s3://covid-eo-data/xco2/*mean*
• delete s3://covid-eo-data/xco2/*diff*
• delete these files from s3://covid-eo-data/BM_500M_DAILY/
• delete s3://covid-eo-data/CropMonitor*

[olafveerman]

This and more was cleaned up.

[Schpidi]

@olafveerman following from the sidelines, is there any change needed for the trilateral dashboard?

[lubojr]

Our observations:
We have updated the CropMonitor urls, which stopped working on trilateral dashboard and now do. But NO2 data for 2020-10-01 are transparent, while NO2 diff has data.