Implement Job splitter and manager in GFMAP

[GriffinBabe]

Implement a job splitter and a job manager in OpenEO GFMAP:

• The Job Splitter takes in parameter a dataset containing polygons/points extraction location and dates. It first separates the job by the temporal-extent and then by the spatial-extent (by default, the spatial split is performed using the s2-grid, to avoid launching jobs that are too large on OpenEO)
• The Job Manager launches jobs and keep track of their progress. If a job fails, logs are saved to a location and the user will be notified of the location of those logs. If the job is successful, the products are downloaded and a post-job script is run locally (if the user specifies it).

Maybe, if too many OpenEO jobs fail then cancel all the other jobs and abort the task.

[kvantricht]

@VincentVerelst you can find an example of dataset chunking here.

[kvantricht]

• Resolution 4 in https://h3geo.org/docs/core-library/restable/
• Crucial to make sure the splitter takes into account the UTM zones.

[GriffinBabe]

@kvantricht From the dataset provided in issue #40 here is a first result from using the h3 library. Seems pretty good to me except maybe for the hexagons including a few points

If I select only the points with extract=True field

[GriffinBabe]

@soxofaan I would also be interested in your opinion on this method as you might be more experienced with the technical limitations of OpenEO. Each hexagon here has an approximative 30km x 30km diameter

[GriffinBabe]

@kvantricht can I use the dataset you provided me as a unit test resource?

[kvantricht]

Looks pretty good indeed. I don't think the ones with just few points are a real problem at the moment? Might be a bit of overhead but seems not a priority to address that, do you think?

[kvantricht]

@kvantricht can I use the dataset you provided me as a unit test resource?

in principle yes but isn't it a bit big for that?

[soxofaan]

@soxofaan I would also be interested in your opinion on this method as you might be more experienced with the technical limitations of OpenEO. Each hexagon here has an approximative 30km x 30km diameter

That looks fine, I don't expect a problem here

[GriffinBabe]

Good
For the unit tests I will upload a subsampled version of it @kvantricht

[GriffinBabe]

Implemented and about to be merged #45