Skip to content

Migration activity support page

Jump to bottom
Daniel Santillan edited this page Dec 7, 2023 · 6 revisions

This page tries to summarize all relevant aspects in relation to migrating the previous manual data configurations of the RACE, EO Dashboard and GTIF instances into a the STAC Catalog generation configuration files present in this repository.

Coordination sheet

To allow coordination between all parties in the migration activity a spreadsheet has been created here.

If you do not have yet access and want to be involved, please request access to the document. The spreadsheet hast a tab per instance. Each tab contains indicators extracted from the configurations that are being migrated. On the left the original identifier and name have been added, the STAC collection Identifier is the new collection identifier and filename of the yaml collection configuration file that can be found in the collections folder of this repository.

Configuration files

EOX makes sure that the placeholder configuration files are available, all identifiers that have a STAC collection Identifier have a corresponding configuration file. These configuration files already have the Resources section filled, which allows extracting the data metadata on catalog generation.

The focus of the review by our partners should be updating and filling following configuration parameters:

(For more information please have a look at the wiki)

Here is a "gold example" configuration file where we tried to consolidate the parameters that should ideally be present. Once a review of one configuration file for these parameters has been done the date when this was done can be added to the Data Sanitation Done of the specific entry in the spreadsheet.

Another task, proposed to be done by the agency, is to review the Title parameter of the configuration file, as this will be displayed in the eodash instance where it is loaded. Once the Title has been reviewed for a collection configuration file it can be marked in the Reviewed Title column of the spreadsheet.

Assets

The configuration files point to the eodash-assets catalog contained data for the Description parameter. All entries in the spreadsheet where the Assets Reviewed column have been filled have a linked markdown file taken from the original configuration of the instances.

It has been decided that the information provided in the markdown files will be reduced to contain only relevant information to the data. Information that tell a story around the data is still of interest and it will be used as basis to create dedicated stories to which the data can point to. The proposed workflow is for someone appointed by the agency to go over the markdown files and delete all non data specific information (in a separate branch) and then create a pull request to main. In the pull request it will be possible to see and track all deletions also after being merged. This would help as basis to decide on creation of new stories in the future. Once a markdown asset file for a collection has been reviewed and edited, a date to the according cell of the Reviewed Description column of the spreadsheet should be added. Still, in order to make sure all markdown files can be explored in their original form a tag was created for the current state, allowing to view the state before deletion at any time: https://github.com/eurodatacube/eodash-assets/tree/descriptions-before-split ( also just in case the same tag was created for the eodash-catalog repository)

Testing of catalog

The race and trilateral test instances showing the catalog are here:

(Please be aware that as testing instances availability is not guaranteed and changes are to be expected)

When creating a pull request to main branch in the eodash-catalog repository the catalog will be build as preview. It is then possible to test the client with preview catalogs using the catalog key in the url:

(Please be aware that as testing instances availability is not guaranteed and changes are to be expected)

It is also possible to see the instance catalog using a generic STAC client, such as STAC Browser:

Evolution

The current configuration files have room for evolution, the idea is to make as much use of potential STAC related evolutions, such as new or updated STAC extensions to provide as much valuable information for description of the data as possible. An issue investigating possible extension and information of interest is here. This issue links to another spreadsheet reviewing possible parameter and extension mapping.

Some aspects / features that have been discussed follow. It might be good to have a separate issue for this or maybe include it in the issue mentioned above.

  • when generating catalog review how/if provide last modified information
  • evaluate how eodash stories the collection is featured in can be linked (currently probably as asset)
  • extend Service definition to allow linking to notebook examples

Parameter mapping to eodash

Here is now a quick rundown of how the parameters map to the client without the new data panel concept, it is important to consider that this is currently evolving and will provide more information from the configuration/catalog files.

image

  • Section 1: provides filter options based on the parameters "Tag", "Satellite", and "Sensor" that is set in the collection yaml file (so each parameter shows a list of all the values available within the category)
  • Section 2: "Country" and "City" are extracted from values present in geodb indicators in the "city" and "country" fields of the tables. Additional values can also come from the "Locations" configuration in the yaml file (special case for "global" layers that only have data on specific locations)
  • Section 3: is the indicator result list where the "Title" of each configuration file is shown.
  • Section 4: when hovering over a poi the "city" value of geodb "city" entry is shown (we could consider adding some other information here in the future)
  • Section 5: shows the "Title" of the yaml configuration file. (When selecting something on the list in section 3, the selected indicator and the title will alway coincide)
  • Section 6: When available shows the "city" attribute of geodb which matches the information of the hover on a point of the map (global datasets do not show information here). Below section 6 is then a chart (when available) and below that the corresponding markdown file that is configured in the "Description" parameters of the yaml file. Shortly to this parameter, it is possible to directly use text there (for example if only very little information is available) or to set a path to a markdown file in the eodash-assets repository.
Clone this wiki locally