chunking schema for analysis-ready?

[Lkruitwagen]

Hey guys! Love to see this effort.

Quick question about the chunking schema you're planning for an analysis-ready corpus. Will you keep the native ERA5 chunking, i.e. {'time':1}? Are you chunking variables together?

With h2ox we've chunked hourly ERA5-land in blocks of 4-years with some moderate spatial aggregation, e.g. 5 degrees. Our main use case is for quick/easy retrieval of timeseries. Would be good to know what you guys are thinking for the chunking schema here!

[alxmrs]

Sorry for the quite late reply. We're releasing data chunked at 1 hour and don't have plans to provide spatial chunks -- though, that is a good idea for the problem you have.

In the medium-term: The time-series use case is an important one that we want to address. We've just updated our roadmap with respect to this goal (#48). To address this, we plan on mirroring our ERA5 data into Google BigQuery (focusing on the AR corpus).

We're planning on using this piece of infrastructure for the data ingestion: https://github.com/google/weather-tools/tree/main/weather_mv#weather-mv-bigquery

[alxmrs]

We did do an internal chunking experiment where we prioritized querying by time. This revealed the inherent Zarr-specific tradeoffs where you have to prioritize between space and time (or, between dimensions A vs dimensions B). Given the other use cases we want to support for this dataset, our plan is to prefer the existing chunking scheme (the whole globe at every hour) and to support timeseries like analysis with BigQuery.

[Lkruitwagen]

Thanks for the update. Glad to see this project going somewhere!