Clarification on gCDM implementation of gridded datasets

[rmcsqrd]

Hi @JohnLCaron @lesserwhirls,

I am trying to understand the motivation behind the differences in the gcdm_grid.proto and gcdm_netcdf.proto files. I am working on adding gridded data functionality to my pygcdm library and am confused with all of the projection/grids/etc message definitions.

The gcdm_netcdf.proto message definitions seemed to have obvious analogs with the CDM data access layer object model. The gcdm_grid.proto message definitions CDM analogs are less clear to me. I see some parallels in the CDM coordinate system object model but when I look at some gridded data (e.g. here) the netCDF file doesn't have any attributes or variables that are directly related to projection. Based on my knowledge of how to request data using messages defined in gcdm_netcdf.proto I don't understand how the user interacts with the projection and transform message definitions in gcdm_grid.proto.

I notice that both gridded message request types (GridDatasetRequest and GridDataRequest) are defined such that the location is a string (similar to the HeaderRequest and DataRequest messages defined in gcdm_netcdf.proto). Is there documentation on what form this location takes in the gridded case? For the gcdm_netcdf.proto files I understand the location string takes this form. If the location string for gridded data is similar then I do not understand how the user accesses different projections or transforms. Could you please clarify this and/or point me in the right direction documentation-wise? Thanks.

[JohnLCaron]

Hey Rio:

Not sure what version you're referencing, but gcdm has evolved into whats in the gcdm module on the development branch (currently version 8): https://github.com/Unidata/netcdf-java/tree/develop/gcdm.

Apologies that the documentation hasnt kept up. This is the first Ive heard that Unidata is interested in gCDM for grids.

gcdm_grid.proto allows coordinate-based access to gridded data. This can scale up to large collections of, eg, GRIB model data. gcdm_netcdf.proto, in contrast, gives remote access to NetcdfDatasets, which has indexed-based access to "cdm datasets" like netCDF.

You should look at the client code in ../develop/gcdm to see the GcdmGridDataset as a gRPC adaptor for ucar.nc2.grid.GridDataset, which is the API. Meanwhile GcdmNetcdfFile is a gRPC adaptor for ucar.nc2.datset.NetcdfDataset, which is the API.

The obvious thing you might want to do in Python is to define analogous APIs for python for NetcdfDataset (which youve probably already done?) and for GridDataset (note this is ucar.nc2.grid.GridDataset, not the older ucar.nc2.dt.GridDataset, which has been removed), and then create adapters that use gRPC to implement those APIs.

You will want to set up a GcdmServer using the java implementation, you dont want to replicate that in python, at least not at first.

You should also note that GcdmServer is not ready for use on unsecured networks. The location string is some minimum string to get things working and tested. There will be another round of design to define all that when we understand how we want to use gCDM. There are example working code in ./develop/gcdm/src/test that you could replicate in python.

John