Experiments for ESIP Presentation

[rajadain]

ESIP has a meeting in July 19-22 https://www.esipfed.org/meetings

Prepare a demo of our work for this meeting.

[lewfish]

A high-level plan for getting something presentable at ESIP. We should aim to get this all working locally first on a tiny subset of the dataset if possible.

• Pick a query that Fernando suggested, and make a concrete instantiation of it. To start, we should pick a time range that either uses the reanalysis OR the prediction dataset.
• Lewis / Terence: Write abstract. See Write abstract for ESIP #49
• Vijay: Save an extract of NHD to S3 to avoid needing database. See Save extract of HUC8 #48
• (Lewis) Rechunk Zarr-encoded data and run benchmarks. See Rechunk Zarr-encoded data and run benchmarks #66
  • Time the query using the Zarr-encoded data that is on S3.
  • (Not sure if we should do this) Time the query using the naive approach, ie. download NetCDF files and do the calculation. This will be slow because we are downloading huge amounts of irrelevant data. I think we want to do this as a baseline, but not sure.
  • Rechunk the data and save it on S3. Not sure, but I think doing it for the whole dataset is going to require a huge amount of resources. We should do it for a large, but manageable subset of the dataset. It needs to be large enough to make the benchmarks "realistic." This corresponds to Task 1-1: Encode NWM Gridded Output as Zarr #26.
  • Time the query using the rechunked Zarr-encoded data. This corresponds to Task 1-2: Benchmark HUC Query of NWM Gridded Zarr Output #27.
• (Terence / Vijay): Reformat data in Parquet and benchmark. See Reformat data in Parquet and benchmark #67
  • Re-encode the data using Parquet for point-based datasets (ie. the ones representing stream reaches). Try different ways of chunking it. This corresponds to Task 1-3: Encode NWM Streamflow Output as Parquet #28. Code in https://github.com/azavea/noaa-hydro-data/blob/master/src/notebooks/save_nwm_sample.ipynb might be a useful starting point.
  • Time the query using the Parquet-encoded data.
• Repeat the queries using different numbers of workers to test scalability.
• Repeat the queries using (interesting) variations on the query or another query that Fernando suggested.
• Display the timings for the different approaches in a table or graph

[lewfish]

Hey @vlulla I’m using ZenHub in GitHub, click this link to join my workspace and see other features available in GitHub or download the ZenHub extension and sign up with your GitHub account.
Posted using ZenHub

[lewfish]

Note that we should keep experiments small at first and use them to estimate how much the full set of experiments will cost and pass that on Stephanie.

[lewfish]

Required reading: Why is Parquet/Pandas faster than Zarr/Xarray here?