Consolidating existing stores?

[jbusecke]

Since we are currently not performing consolidation (waiting for pangeo-forge/pangeo-forge-recipes#575), we have two options for the future:

• go through all the stores manually and consolidate them
• reprocess them one more time once the above PR is done. (this might be cool once we move the target location to the old Google Bucket).

[cisaacstern]

I think there's a third option of automated use of linked PR against existing stores?

Where the input PCollection is just the zarr.storage.FSStore of the existing store.

[cisaacstern]

Also pangeo-forge/pangeo-forge-recipes#556 falls into this category.

[cisaacstern]

I think there's a third option of automated use of linked PR against existing stores?

Where the input PCollection is just the zarr.storage.FSStore of the existing store.

I think this is what I will do for the ClimSim mlo data leap-stc/ClimSim#38 (comment) which is slow to load without pangeo-forge/pangeo-forge-recipes#556.

[jbusecke]

I am afraid I do not quite understand what that third option is?

[cisaacstern]

I am afraid I do not quite understand what that third option is?

Run a pipeline like this on Dataflow:

from pangeo_forge_recipes.transforms import ConsolidateCoordinateDimensions

existing_paths: list[str] = get_existing_paths_from_bigquery(...)

def path_to_fsstore(path: str) -> zarr.storage.FSStore:
    ...
    return store

recipe = (
    beam.Create(existing_paths)
    | beam.Map(path_to_fsstore)
    | ConsolidateCoordinateDimensions()
)

[jbusecke]

Ahhhhh, yes that makes sense. I could do that in retrospect once, and then add such a stage to new recipes.

[jbusecke]

Still relevant. I am copying the successful ingestions over to the public buckets and catalog them in leap-pangeo.cmip6_pgf_ingestion.leap_legacy. We could probably run a script over this and consolidate the coordinates afterwards.

[jbusecke]

Just going through old issues. I think this might actually be addressed by our current QC (i.e. unconsolidated stores are not passing the tests?), but would need to check that