Perform preprocessing to geoparquet from patch extractions

[kvantricht]

Starting from patch extractions, we need a workflow that performs:

• loading extractions as openEO datacubes
• perform preprocessing
• perform pixel sampling
• export result as geoparquet

[VincentVerelst]

Currently it's not possible on the CDSE backend to load_stac multiple CRS: Open-EO/openeo-geopyspark-driver#827

[VincentVerelst]

Implemented a STAC splitter in gfmap to circumvent the load_stac issue: Open-EO/openeo-gfmap#146

[kvantricht]

Needs #125 first.

[jdries]

moving to public mount is possible, but this would trigger a huge update in stac metadata

[VincentVerelst]

• Successfully loaded precomposited METEO from STAC on Terrascope; this gives a huge performance boost
• Was able to generate a parquet file with preprocessed data of meteo, dem, s1 and s2 timeseries, using some random points
• /vitodata/worldcereal_data should be available on openeo-dev.vito.be soon (waiting on integration tests). This would mean we can load METEO and the Sentinel 1 and 2 patches from STAC from there
• For interaction with the RDM API we are still dependent on two issues, see: Setup ground truth geoparquet generation pipeline using RDM API #125

[kvantricht]

That's great progress Vincent, thanks! So once the RDM interaction is there we have in principle everything in place to start gradually building this geoparquet and performing some downstream tests on them?

[VincentVerelst]

That's great progress Vincent, thanks! So once the RDM interaction is there we have in principle everything in place to start gradually building this geoparquet and performing some downstream tests on them?

Indeed!

[kvantricht]

Conceptual pipeline:

• per ref_id per EPSG zone, get temporal_extent from STAC metadata filtered on the ref_id specific patches
• RDM interaction module gets geometries from RDM based on BBOX of all patches in the job and the temporal extent
• To be implemented: some way to distinguish original seed features vs collateral for downstream tracking (@cbutsko )
• Resulting geodataframe is written to geoparquet, containing all required attributes, to temporary folder on artifactory
• This geodataframe immediately is used (load_url) to launch patch-to-point script with same AOI and temporal extent
• Final result is downloaded as separate geoparquet
• To be implemented: some post_job_action that allows to merge this result safely into the global geoparquet

[VincentVerelst]

Blocked by gdalerrors when using load_stac to load in patch extractions