Train with Digital Elevation/Surface Models

[dokterbob]

For many applications of earth observation, elevation data as well as derived metrics is essential.

I noticed that v0 was trained with GLO90. I am wondering why this was left out in v1.

It would be great to have v2 trained with much higher resolution datasets when training on high-res ortho imagery and to train with GLO30 otherwise. This way, the model should have a general idea of elevation.

Dataset suggestions

• USGS 3DEP https://www.usgs.gov/3d-elevation-program
• New Zealand Elevation https://registry.opendata.aws/nz-elevation/

[brunosan]

Thanks @dokterbob.
That's correct, v0.2 used DEM and v1 does not.
The technical reason is that on v1 we parametrize all inputs according to the sensor wavelength. This is the only way we could think to make Clay agnostic of instrument.

Alas, this means we can't add DEMs on that framework. This is a known regression, and something we are thinking about adding on next versions. May be we can have some optional fixed channels, like DEMs. We would need to ensure that the resolution is also passed to the self-attention, and we take care of resolution missmatches with the bands.

PS: One of the test I'd like to run is to regress v1 embeddings to DEM. I have the hunch that DEMs are implicitly learned to some degree. After all, we humans can roughly guess the dem looking at an image.

[rbavery]

dealing with differences in resolution can be pretty tricky. Some new architectures for this have been proposed, but unfortunately haven't been used in large visual foundational models like PaliGemma 2 or Segment Anything yet. But maybe these would be architecture ideas to incorporate in a v2 Clay model?

NaVIT: https://arxiv.org/pdf/2307.06304
huggingface/pytorch-image-models#1952

FlexiVIT: https://arxiv.org/abs/2212.08013
https://x.com/dinkar_juyal/status/1790463209406935185