EWoc Data Access Gateway

Data access (download, search, upload) for EWoC processors based mainly on s3 buckets. Optionaly EOdag for search and download can be used.

Data downloading

Currently you can acess EO data from the following sources:

Data Provider	Sentinel-2 L1C	Sentinel-2 L2A	Landsat 8 C2 L2	Sentinel-1 GRD
AWS buckets	Yes 1	Yes 1, 2	Yes 1	Yes 1
Creodias DIAS bucket 4	Yes	Partially 3	No	Yes
Creodias Finder 5	Yes	Partially 3	No	Yes

Currently you can acess DEM data from the following sources:

Data Provider	SRTM 1s	SRTM 3s	COP DEM 1s	COP DEM 3s
EWoC private bucket	Partially	Yes	No	No
AWS buckets	No	No	Yes	Yes
Creodias DIAS bucket 4	Yes	No	Yes	Yes
ESA website	Yes	No	No	No

¹ Access payed by requester.

² L2A cogs are available.

³ Rolling archive of 1 year.

⁴ Creodias bucket can be acessible only from creodias network

⁵ Through EODAG

EWoC Data search

You can list the data from the EWoC ARD and Aux data private buckets by scanning the bucket from a specific prefix.

EWoC Data upload

You can upload data to the EWoC private buckets: ARD or PRD.

Installation

1. Retrieve the package
2. Create a venv
3. pip install /path/to/ewoc_dag.archive

Usage

Credentials

To manage access to the different data provider or buckets you need to set some credentials:

Data Provider or buckets	Credentials
EWoC private bucket on CreoDias	Set env variables: EWOC_S3_ACCESS_KEY_ID and EWOC_S3_SECRET_ACCESS_KEY
Creodias DIAS bucket	N/A but access possible only from creodias cluster
AWS buckets (public or private)	Set env variables: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY 6
ESA website	N/A but access not useable for production
EODAG	cf. EODAG documentation

⁶ Other methods are described here

Special environment variables

It is possible to control some other points through environment variables:

• EWOC_CLOUD_PROVIDER is the env variable which indicates where the processing system is deployed: aws or creodias are currently supported. This env variable is used to define the default provider of the processing system for Sentinel-2, Copernicus DEM and Sentinel-1 data. The idea is to use the buckets of the cloud provider.
• EWOC_COPDEM_SOURCE is the env variable which indicates which provider is used as default to retrieve Copernicus DEM data: aws or creodias or eodag
• EWOC_S1_PROVIDER is the env variable which indicates which provider is used as default to retrieve Sentinel-1 data: aws or creodias or eodag
• EWOC_S2_PROVIDER is the env variable which indicates which provider is used as default to retrieve Sentinel-2 data: aws or creodias or eodag
• EWOC_EODAG_PROVIDER is the env variable which indicates which EODAG provider you want to use to retrieve data with EODAG.
• EWOC_DEV_MODE is the env variable which indicates to use the EWoC dev buckets (ARD or PRD) to upload data

CLI

Two cli are provided:

• the first one ewoc_get_eo_data provided the EO data needed by the EWoC processors based by their product ID.
• the second one ewoc_get_dem_data provided the dem data needed by the EWoC processors based on the S2 tile ID.

Python

If you want to acess to EO data from other python code you can use directly the get functions in the python module. For example:

from ewoc_dag.landsat8 import get_l8c2l2_product

get_l8c2l2_product('l8_c2l2_prd_id')

❕ Don't forget to set crendentials associated to the provider used.

If you want to upload data to the EWoC buckets:

from pathlib import Path
from ewoc_dag.bucker.ewoc import EWOCARDBucket

ewoc_ard_bucket = EWOCARDBucket(ewoc_dev_mode=True)
ewoc_ard_bucket._upload_file(Path('/tmp/upload.file'),'test.file')
ewoc_ard_bucket._upload_prd(Path('/tmp/upload_test_dir'),'test_up_dir')

❕ Don't forget to set EWoC bucket credentials

If you want to list the content of the some part of the ARD bucket and save as a satio collection:

from ewoc_dag.bucket.ewoc import EWOCARDBucket

ewoc_ard_bucket = EWOCARDBucket(ewoc_dev_mode=True)
ewoc_ard_bucket.sar_to_satio_csv("31TCJ", "0000_0_09112021223005")
ewoc_ard_bucket.optical_to_satio_csv("31TCJ", "0000_0_09112021223005")
ewoc_ard_bucket.tir_to_satio_csv("31TCJ", "0000_0_09112021223005")

or to list the AgERA5 data

from ewoc_dag.bucket.ewoc import EWOCAuxDataBucket

ewoc_auxdata_bucket = EWOCAuxDataBucket()
ewoc_auxdata_bucket.agera5_to_satio_csv()

❕ Don't forget to set EWoC bucket credentials

How to release

TODO