Opening NetCDF makes computer crash

[EmileSonneveld]

What is the bug or the crash?

When I open this file with QGIS, my whole user session crashes. I get a black screen and need to log in again.
I think it is due to an over de top amount of RAM being allocated:
https://emilesonneveld.be/misc/Sentinel2_makes_linux_user_session_crash.nc (Save any work before opening this NetCDF)

Steps to reproduce the issue

1. Open the attached file: https://emilesonneveld.be/misc/Sentinel2_makes_linux_user_session_crash.nc
2. Wait
3. Experience a crash

Versions

3.22.4-Białowieża

Supported QGIS version

• I'm running a supported QGIS version according to the roadmap.

New profile

• I tried with a new QGIS profile

Additional context

I made a new profile and added the standard OpenStreetMap as background. Still got the crash
I run Ubuntu with kernel 6.2.0-39-generic
My colleague runs Linux mint and has the same problem

[agiudiceandrea]

@EmileSonneveld, thanks for reporting. QGIS 3.22 is no longer supported. May you please actually try if the issue occurs also using a currently supported QGIS version (QGIS 3.34.2, QGIS 3.28.14) and provide the full version info ("In the QGIS Help menu -> About, click in the table, Ctrl+A and then Ctrl+C. Finally paste here. Do not make a screenshot.") as requested in the issue report form?

[rouault]

The huge RAM consumption arises when opening the file with "Add mesh layer" . It works fine with "Add raster layer" (using GDAL netCDF driver)

[rouault]

@PeterPetrik Debugging this, I see QGIS spends its time in MDAL::DriverGdal::initFaces(). On this dataset where width = 16831 and height = 12992, and my understanding of "typedef std::vector Faces;" and "typedef std::vector<size_t> Face;", my estimate of the minimum amount of RAM required is 16830 * 12991 * (sizeof(size_t) + sizeof(void*) + 4 * sizeof(size_t)) ~= 10 GB, and probably more due to memory fragmentation given we allocate lots of Face elements one at a time.
As adding piece-wise loading of data is probably a difficult undertaking, it seems that MDAL should bail out on a GDAL dataset beyond some dimension, likely correlated to available RAM (there's a GDAL CPLGetUsablePhysicalRAM() function for that)

[EmileSonneveld]

Loading it without the mesh indeed avoids the problem! Can I disable mesh creation by default in QGIS? Or configure the NetCDF to avoid mesh creation?
The ram usage could indeed be 10Gb+. That's about the amount that was free at time of crash.

[PeterPetrik]

Hi and thanks for report and debugging: created MDAL issue for this lutraconsulting/MDAL#476

[EmileSonneveld]

In 3.34.2-Prizren, only QGIS crashes without error message. Which is quite better. It allows to try again without the mesh

[JaqucesT]

Opening this file in 3.34.3 memory consumption rises till 36 GB in steps and then it starts rising in a linear way as if there is a memory leak. When QGis reaches 50 GB of memory consumption i get a "Bad allocation" error, but the program stays alive. Removing this layer then takes a long time where you can see memory consumption drop to some normal 350 MB.

[rouault]

Another instance of that issue in #56786 (comment)