New hdf5/netcdf4 compression

[JohnLCaron]

ed hartnett is working on adding new compression to netcdf4. In particular, zstandard is ready to roll.

more generally, the hdf5 group reports: "Currently we provide decoders for BZ2, BLOSC, LZ4, LZF, ZFP, and MAFISC."

see Unidata/netcdf-c#1545 for more.

[lesserwhirls]

I believe @haileyajohnson is on it. Compression is also an issue with Zarr support, and I believe she is looking at creating a codec subproject to handle the various compression schemes (refactoring our current HDF5 compression related code out of core and into the codec subproject). The tricky part is finding pure java encodes/decoders to work with. There are some promising candidates, but they are not fully worked out, or they "secretly" rely on transitive native code.

[JohnLCaron]

Sorry, I just saw this. Hailey, let me know how youd like to proceed. I have an initial implementation in a PR I just sent. Feel free to refactor it as needed.

[haileyajohnson]

So what you have is great for the develop branch, but I've fully refactored filters in 5.x, so that will take some work to port. But the heavy work of decompressing will already be written, which is great :)

[JohnLCaron]

https://support.hdfgroup.org/HDF5/doc/H5.format.html

has:

Identification	Name	Description
0	N/A	Reserved
1	deflate	GZIP deflate compression
2	shuffle	Data element shuffling
3	fletcher32	Fletcher32 checksum
4	szip	SZIP compression
5	nbit	N-bit packing
6	scaleoffset	Scale and offset encoded values

of which netcdf-java only has deflate, shuffle and fletcher32.
last known, szip had IP issues.

http://www.hector.ac.uk/cse/distributedcse/reports/nemo/nemo_notes/node56.html

has

There are a number of different filters which can be applied to an HDF5 dataset:
N-bit filter - essentially compresses the data by removing the unused bits before storing the data on output. The data is then unpacked on input restoring the missing bits. Quite complex to use but may save diskspace. Checks would be required such that no information is lost from the data.
Scale-offset filter - performs a scale and offset on each data value truncating the resulting value to a fixed number of bits before storing. E.g the operation performed is $d_{new} = a*d_{old}(i) + b$ where $d_{old}$ is the original data value, $d_{new}$ is the new data value, $a$ is the scale and $b$ is the offset. minimum-bits determines the minimum number of bits that will be used. For integer data the filter is lossless (unless too small a value for minimum-bits is selected. However, for floating point data, the filter translates the floating point data to integer data (the filter is lossy - information is lost due to its action) and so is not useful for NEMO.
Szip filter - the Szip compression

SO, if we had some example data, we could try adding the last two.

[haileyajohnson]

We have scaleoffset too, and now there are 3 32 bit checksum filters: fletcher, adler, and CRC. nbit is not in there right now.

[JohnLCaron]

very cool, good work!

[JohnLCaron]

Ive just added a zstandard filter (PR #984)

  public enum FilterType { none(0), deflate(1), shuffle(2), fletcher32(3), szip(4), nbit(5), scaleoffset(6),
    zstandard(32015), unknown(Integer.MAX_VALUE);

[JohnLCaron]

we could also ask HDF about BZ2, BLOSC, LZ4, LZF, ZFP, and MAFISC. Are they documented somewhere?

[rschmunk]

A possibly clueless question, but will these projects affect netCDF-Java support for HDF5 1.10.5? I've had a recent query from a group at NASA/GSFC who are planning a data product, and Panoply, IDV, and other apps using NJ cannot open their test files due to lack of such support.

[haileyajohnson]

In the short term, we're only looking at adding zstandard support, so if the reason they can't open the file is that it was compressed with ztandard, yes, otherwise no. The main goal here is to ensure that files written with the netcdf-c library can be read using the netcdf-java library. Building up to include every filter that hdf5 supports is a longer road.

[JohnLCaron]

Ill let Hailey answer your questions, but we'd love to get any test files with compression problems.

[rschmunk]

@JohnLCaron, The problem I am having is that the dataset is in HDF5 1.10.5 and cannot be opened due to an exception, with msg "Unknown superblock= 3". Discussion in the Unidata/thredds repo back in 2016 indicates this is due to a lack of support for that version of HDF 1.10.x. At the end of that discussion, there is indication that support for HDF5 1.10 was added to NetCDF-C in 2017.

[JohnLCaron]

Ok, thanks for explanation. My intention is to eventually add support for 1.10. So gathering example files (esp small ones eg a few mb) will be useful for testing that.

[rschmunk]

I'll check with the developer at GSFC to see if it's okay to share.