NullPointerException reading from NetcdfFile when cache enabled (version 5.5.3)

[skauppin]

Hi all,

I'm working with a multithreaded server that uses netcdf-java library to access netcdf files and providing data from them to client applications. We recently updated from netcdfAll-4.6.10.jar to version 5.5.3 (edu.ucar/cdm-core, edu.ucar/grib and edu.ucar/netcdf4).

We are utilizing the caching functionality of the netcdf-java library. Then the system starts it initializes the cache by calling NetcdfDatasets.initNetcdfFileCache, and we get a NetcdfFile instance by calling NetcdfDatasets.acquireFile.

Now with version 5.5.3 we are observing errors when reading data from files.

java.lang.NullPointerException: null
        at ucar.unidata.io.RandomAccessFile.read_(RandomAccessFile.java:737)
        at ucar.unidata.io.RandomAccessFile.readBuffer(RandomAccessFile.java:503)
        at ucar.unidata.io.RandomAccessFile.seek(RandomAccessFile.java:491)
        at ucar.nc2.iosp.hdf5.DataBTree$Node.<init>(DataBTree.java:163)
        at ucar.nc2.iosp.hdf5.DataBTree$DataChunkIterator.<init>(DataBTree.java:128)
        at ucar.nc2.iosp.hdf5.DataBTree.getDataChunkIteratorFilter(DataBTree.java:67)
        at ucar.nc2.internal.iosp.hdf5.H5tiledLayoutBB.<init>(H5tiledLayoutBB.java:97)
        at ucar.nc2.internal.iosp.hdf5.H5iospNew.readData(H5iospNew.java:222)
        at ucar.nc2.internal.iosp.hdf5.H5iospNew.readData(H5iospNew.java:200)
        at ucar.nc2.NetcdfFile.readData(NetcdfFile.java:2122)
        at ucar.nc2.Variable.reallyRead(Variable.java:797)
        at ucar.nc2.Variable._read(Variable.java:736)
        at ucar.nc2.Variable.read(Variable.java:614)

When not using the cache functionality there is no errors but then we have some performance problems.

To me it looks like the problem is that NetcdfFile is be removed from the cache (and closed) while it's still used by some thread.

    // System startup 
    NetcdfDatasets.initNetcdfFileCache(...);


    // Then opening files and reading data (multiple threads processing files in parallel)
    try(NetcdfFile ncFile = NetcdfDatasets.acquireFile(...)) {
        
        Variable var = ncFile.findVariable(variableName);
        
        // cache eviction happening for the file the thread is handling
        
        // trying to read variable data --> java.lang.NullPointerException
        Array data = var.read().reduce();
    }

This did not happen with library version 4.6.10.

We haven't tried yet, but I guess a kinda workaround could be to set the cache size so big that all the files being processed probably fit to the cache. That's of course not optimal in anyway, and we'd prefer a more sound solution.

Reproducing the problem seems quite easy. Just open and read from X files in parallel and while setting cache size < X.

I would appreciate any comments, insight, ideas how this should be fixed.

Thanks!
Sami

[JohnLCaron]

Thanks for the report, Sami. Is it possible to get the server logs when this happens?

[haileyajohnson]

Hi @skauppin , older support messages show that this has been an issue before, and it is a bug in netCDF-Java, but it can be mitigated by also enabling the RandomAccessFile cache: RandomAccessFile.enableDefaultGlobalFileCache();

Let us know if that doesn't work for you.

[skauppin]

Thanks for the answer!

I was able to reproduce the original problem on my laptop and enabling the RandomAccessFile cache did seem to help. But when we deployed to production environment we still got those NullPointerExceptions errors from RandomAccessFile. Many files are being accessed there in multiple threads so it's more complicated.

I haven't been able to reproduce on my laptop with RandomAccessFile cache enabled, so at the moment I'm not able to provide more details on what is happening.

I'll let you know if we find out anything.