Add maap biomass datasets to the API (high-level steps)

[abarciauskas-bgse]

For each dataset, we will follow the following steps:

Identify the dataset and what the processing needs are

1. Identify dataset and where it will be accessed from.

Datasets in https://earthdata.nasa.gov/maap-biomass/ need to be published to the VEDA API so that users, such as the trilateral dashboard, can access them. We will also need these datasets in the NASA dashboard once we are ready to publish the biomass story.

These datasets are in https://github.com/MAAP-Project/biomass-dashboard-datasets/tree/main/datasets, presumably we want to publish all of those datasets to the staging API, but we should probably cross check with the biomass story being told for the trilateral dashboard and focus on the ones we are aware of.

Datasets were often required to be uploaded to a landing zone location, but we will have to go through each one to identify its location in the MAAP buckets, I believe most of them are in s3://maap-landing-zone-gccops/user-added/uploaded_objects/

Question for the group will be if we want to copy the files to our "VEDA" bucket. Who should be able to access these files?

3. If the dataset is ongoing (i.e. new files are continuously added and should be included in the dashboard), design and construct the forward-processing workflow.
   • Each collection will have a workflow which includes discovering data files from the source, generating the cloud-optimized versions of the data and writing STAC metadata.
   • Each collection will have different requirements for both the generation and scheduling of these steps, so a design step much be included for each new collection / data layer.

Design the metadata and publish to the Dev API

1. Review conventions for generating STAC collection and item metadata:

   • Collections: STAC Collection Creation Conventions (Dashboard Specific) veda-backend#29 and STAC version 1.0 specification for collections
   • Items: STAC Item Creation Conventions (Dashboard Specific) veda-backend#28 and STAC version 1.0 specification for items
   • NOTE: The delta-backend instructions are specific to datasets for the climate dashboard, however not all datasets are going to be a part of the visual layers for the dashboard so I believe you can ignore the instructions that are specific to "dashboard" extension, "item_assets" in the collection and "cog_default" asset type in the item

2. After reviewing the STAC documentation for collections and items and reviewing existing scripts for generating collection metadata (generally with SQL) and item metadata, generate or reuse scripts for your collection and a few items to publish to the testing API. There is some documentation and examples for how to generate a pipeline or otherwise document your dataset workflow in https://github.com/NASA-IMPACT/cloud-optimized-data-pipelines. We would like to maintain the scripts folks are using to publish datasets in that repo so we can easily re-run those datasets ingest and publish workflows if necessary.

3. If necessary, request access and credentials to the dev database and ingest and publish to the Dev API. Submit a PR with the manual or CDK scripts used to run the workflow to publish to the Dev API and include links to the published datasets in the Dev API

Publish to the Staging API

Once the PR is approved, we can merge and publish those datasets to the Staging API

[abarciauskas-bgse]

Each of these data products was uploaded to the MAAP bucket by users, however those "users" may have been David Minor through some implicit approval from the Biomass Harmonization group

We should check it's ok to duplicate these files in the nasa-maap-data-store and API:

• CCI Biomass https://titiler.maap-project.org/mosaicjson/mosaics/f343756d-bf15-4095-a8f3-f4fcbb26b5f9/mosaicjson
• NCEO Africa s3://maap-landing-zone-gccops/user-added/uploaded_objects/ae841191-8a43-4226-9c5e-db4f500adb45/AGB_map_2017v0m_COG.tif
• GEDI L4B

These 2 products are provisional so we are not going to publish them for now:

• ICESat-2 Boreal Biomass https://titiler.maap-project.org/mosaicjson/mosaics/e7e26f4d-9a2f-4a3b-9256-0821e372e40e/mosaicjson (~4000 items to copy). This we may want to setup S3 replication for
• NASA JPL https://titiler.maap-project.org/mosaicjson/mosaics/f0e6f941-4e50-46b3-9463-11b3c2181662/mosaicjson

I'm assuming we don't need any of the others listed here https://github.com/MAAP-Project/biomass-dashboard-datasets/tree/main/datasets because no one has asked for them

• CCI Biomass standard deviation
• ICESat-2 Boreal Biomass Standard deviation
• ICESat-2 Boreal Biomass inputs: Landsat 8, Topo
• NCEO Africa Standard Deviation
• NASA JPL Standard Error
• Paraguay datasets (Estimated Biomass, Forest Mask, Tree cover)

Next steps

• Configure access from 853558080719 to nasa-maap-data-store This bucket permits public access

For NCEO Africa 2017

• Check it's ok these files are publicly accessible via nasa-maap-data-store
• Copy to nasa-maap-data-store
• publish to dev and staging APIs

For GEDI Gridded Biomass L4B

These files are published from ORNL DAAC, https://cmr.earthdata.nasa.gov/search/concepts/C2244602422-ORNL_CLOUD.html

• Add a dynamic tiler for ORNL cc @anayeaye
• Publish to dev and staging APIs

CCI BIOMASS

• Check with Heather Kay about publishing these data files
• Check it's ok these files are publicly accessible via nasa-maap-data-store
• Copy to nasa-maap-data-store
• publish to dev and staging APIs

ICESat-2 Boreal 2020

this will be updated soon so we won't publish yet

NASA JPL 2020

this is a provisional product we will not publish

[j08lue]

Stale