Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add custom raster data provider for lazy evaluation #1008

Open
14 tasks
janzandr opened this issue Nov 1, 2024 · 1 comment
Open
14 tasks

add custom raster data provider for lazy evaluation #1008

janzandr opened this issue Nov 1, 2024 · 1 comment
Assignees
Labels
feature request New feature or request

Comments

@janzandr
Copy link
Contributor

janzandr commented Nov 1, 2024

Currently we use GDAL VRTs with Python pixel functions for implementing products with lazy evaluation (e,g, spectral indices).
But VRTs do have some limitations:

  • input and output bands must have the same CRS
  • the VRT pixel function interface is not very flexible
    image
  • each output band needs to specify it's own pixel function
  • all the structural metadata like extent, size and CRS need to be specified in the (hand-written) VRT itself
  • we always need to create a VRT file in the file system

A custom QgsRasterDataProvider is way more flexible:

  • we can have a url on the file system storing all required information...
  • ... but we could also provide all required information coded inside the url string itself without any file system operation
  • we can support different CRS for inputs and outputs
  • way better API support for implementing new functionality
  • single processing function handling all output bands at once

It is proposed to implement a LazyEvaluatedRasterDataProvider, that offers several features. E.g.

  • on-the-fly preprocessing of a raster:

    • convert data type
    • translate/warp into new target grid (extent, size and CRS)
    • subset bands
    • data scaling (band offset and scale)
    • interprete raster as a mask (mask values, ranges and bits)
  • on-the-fly preprocessing of multiple rasters:

    • masking a raster with another raster mask
    • raster math, e.g. (r1+r2)/r3
    • raster aggregation:
      • mean/median/min/max/... composite
      • simple mosaicking
    • stack bands from multiple raster source with non-matching CRS! (GDAL VRTs can't do this without warping beforehand)

All the above are just a collection of functionalities that the user should be able to use out-of-the-box. For more complex/custom operations, the user can implement their owns processing functions, which will be evaluated on-the-fly.

  • custom user functions

Finally, we should choose a unique file extension for the provider and register it in QGIS, so that the correct raster layer with our provider is created. Similar to the *.geojson extension for GeoJSON, it is proposed to introduce the *.lerjson or lazyjson extension representing a LazyEvaluatedRasterJSON.

@janzandr janzandr added the feature request New feature or request label Nov 1, 2024
@janzandr janzandr added this to the Future Release milestone Nov 1, 2024
@janzandr janzandr self-assigned this Nov 1, 2024
@janzandr
Copy link
Contributor Author

janzandr commented Nov 1, 2024

The specification of a lazily evaluated raster could look like this:
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request
Projects
None yet
Development

When branches are created from issues, their pull requests are automatically linked.

1 participant