Gain familiarity with zarr

[tjcrone]

Zarr is a storage layer for multidimensional numerical arrays. It allows array data to be chunked, compressed, and read to / written from a range of different storage mediums. Via gcsfs, zarr can store its data directly in google cloud storage buckets. Xarray recently gained the ability to read and write to zarr stores. We will need to figure out how to use zarr in Azure.

The following combination of packages will likely provide an ideal way to work with CAMHD datasets in the cloud: xarray + dask + zarr + gcsfs (or whatever the equivalent is on Azure).

To get spun up on zarr, take the following steps:

• Skim the zarr documentation
• Skim the gcsfs documentation
• Practice creating zarr arrays and storing them to disk
• Practice writing and reading zarr arrays to google cloud storage via gcsfs
• Practice using xarray to write / read datasets to zarr stores on disk
• Practice using xarray to write / read datasets to zarr stores in google cloud storage via gcsfs