Add VIIRS nightlights dataset to the API

[abarciauskas-bgse]

Identify the dataset and what the processing needs are

1. Identify dataset and where it will be accessed from. Check it's a good source with science team. Ask about specific variables and required spatial and temporal extent. Note most datasets will require back processing (e.g. generating cloud-optimized data for historical data).

From https://github.com/NASA-IMPACT/covid-api/blob/develop/covid_api/db/static/datasets/nightlights-viirs.json#L11 it looks like the VIIRS nightlights data is in s3://covid-eo-data/bm_500m_daily/VNP46A2_V011_*_cog.tif

• Check if there are new files
• Determine if new files require conversion to COG

Design the metadata and publish to the Dev API

1. Review conventions for generating STAC collection and item metadata:

   • Collections: STAC Collection Creation Conventions (Dashboard Specific) veda-backend#29 and STAC version 1.0 specification for collections
   • Items: STAC Item Creation Conventions (Dashboard Specific) veda-backend#28 and STAC version 1.0 specification for items
   • NOTE: The delta-backend instructions are specific to datasets for the climate dashboard, however not all datasets are going to be a part of the visual layers for the dashboard so I believe you can ignore the instructions that are specific to "dashboard" extension, "item_assets" in the collection and "cog_default" asset type in the item

2. After reviewing the STAC documentation for collections and items and reviewing existing scripts for generating collection metadata (generally with SQL) and item metadata, generate or reuse scripts for your collection and a few items to publish to the testing API. There is some documentation and examples for how to generate a pipeline or otherwise document your dataset workflow in https://github.com/NASA-IMPACT/cloud-optimized-data-pipelines. We would like to maintain the scripts folks are using to publish datasets in that repo so we can easily re-run those datasets ingest and publish workflows if necessary.

3. If necessary, request access and credentials to the dev database and ingest and publish to the Dev API. Submit a PR with the manual or CDK scripts used to run the workflow to publish to the Dev API and include links to the published datasets in the Dev API

Publish to the Staging API

Once the PR is approved, we can merge and publish those datasets to the Staging API

[xhagrg]

Is this separate from the other nightlight data we have?

[slesaad]

The data has been ingested and following cities/id are available at https://staging-stac.delta-backend.xyz/collections/nightlights-500m-daily/items:

Beijing: VNP46A2_V011_be_<date>_cog
Caracas: VNP46A2_V011_ca_<date>_cog
LosAngeles: VNP46A2_V011_la_<date>_cog
NewYork: VNP46A2_V011_ny_<date>_cog
SanFrancisco: VNP46A2_V011_sf_<date>_cog
EUPorts: VNP46A2_V011_EUPorts_<date>_cog
Tokyo: VNP46A2_V011_tk_<date>_cog