Some UCAR models are broken

[nwagenbrenner]

Broken models include NDFD (error: "Failed to download forecast file. File is not an *.nc file.") and RAP (error: "Geopotential_height_cloud_tops is out of range in forecast file."). Note: semi-related issue #494.

[masonwillman]

Here are my findings from some preliminary testing:

NDFD:

• The error has to do with the url of NDFD. The http fetch request in line 421 in wxModelInitialization.cpp is failing, which triggers a file check later that outputs this error.
• By setting the environment variable CPL_DEBUG=WINDNINJA, we can output the url request for NDFD. This is where we should be looking to fix this error

RAP:

• The http request is working correctly (was able to manually check), the problem is in ncepRAPSurfInitialization.cpp at line 247. This is what is throwing the exception.
• We will have to check to see if clouds exceeding that range is normal, and if it is not, how do we download the correct range of cloud data.

@latwood can use this as a starting point when they have time to work on this.

[masonwillman]

Updates on the models:

NDFD:

• Originally thought the problem was tied to issue #494, so I adjusted the time_duration=USER_TIME to either be 0 or 12 to match the database . However, further tested showed that the downloads were still not working.
• Once we figure out how what is wrong with the url, we can adjust the generation here here
• Environment variable CPL_CURL_VERBOSE=ON useful to see the http request and url being outputted.
• Example url being generated: http://thredds.ucar.edu/thredds/ncss/grid/grib/NCEP/NDFD/NWS/CONUS/NOAAPORT/Best?north=45.0836&west=-116.295&east=-113.275&south=42.724&time_start=present&time_duration=PT6H&accept=netcdf3&var=Total_cloud_cover_entire_atmosphere_single_layer_layer,Wind_direction_from_which_blowing_height_above_ground,Wind_speed_height_above_ground

RAP:

• The problem is that the data being downloaded is extremely large. The values are 9.96...e+34 which are too large for cloud top heights. I confirmed these values using the same variable and lat, long inputs on the server website. I checked other data with other lat long cords, and this is not consistent with all data. My thinking could be that this is a new value that indicates there are no clouds in the sky, but that is a guess.
• Next steps are unclear. We could correspond with the station to confirm this is the correct data, we could account for these large values in our data check, or we can set these large values to 0 as a temporary fix.

@nwagenbrenner it would be nice to provide some input here when you get a chance since I feel as if I am hitting a dead end. Not sure what is wrong with the url and would like input on the RAP data.

[masonwillman]

Contacted the IT support desk, this was their response. This should give us enough information to properly deal with both issues now.

Greetings Mason!

First, let's talk about the NDFD collection. Occasionally GRIB messages come across the NOAAPORT satellite feed that are corrupt or misencoded, and that appears to be the issue with the NDFD. If you visit the main catalog:

https://thredds.ucar.edu/thredds/catalog/grib/NCEP/NDFD/NWS/CONUS/NOAAPORT/catalog.html

You will see that both the FULL and BEST datasets have two links:

• LambertConformal_1377X2145 (Center 38.2296N 95.4386W)
• LambertConformal_1597X2345 (Center 40.6279N 98.3417W)

What that means is that there are two horizontal domains represented by the the collection of GRIB2 messages that the TDS is serving for the NDFD (note: there should only be one). They have a slightly different center-point of the domain, and slightly different grid point spacing. As such, the TDS creates a FULL and BEST dataset for each. If there is only one domain represented by the collection, you will just get one dataset for each type (best, full). Once the second domain showed up, the TDS started producing two of each - so best, on its own, no longer works. This does not happen often, but it does happen.

One strategy would be to try:
http://thredds.ucar.edu/thredds/ncss/grid/grib/NCEP/NDFD/NWS/CONUS/NOAAPORT/Best

first, and if that returns an HTTP 404 error, use
https://thredds.ucar.edu/thredds/ncss/grid/grib/NCEP/NDFD/NWS/CONUS/NOAAPORT/Best/LambertConformal_1377X2145-38p23N-95p44W-2

1377X2145-38p23N-95p44W is the expected domain at this point in time, the other likely comes from a series of corrupted GRIB2 messages. Once the bad GRIB messages age out of the system, the best dataset link (without the horizontal domain info) will start working again.

Next, let's look at the RAP. Unfortunately, I don't see the output attached to this message (maybe it got dropped by our support system). However, I used the link you provided to generate a netCDF 3 file of the subset. What I see in the netCDF file itself is that the missing data are encoded as NaNf. I'm unsure how you generated the CSV output from the netCDF file, but it's possible that is interpreting NaNf as the maximum float value for the system?

Cheers!
Sean

[masonwillman]

Additional information provided by Sean about the NaNf values:

Greetings Mason!

That is correct - a NaNf entry would indicate no cloud top was computed as no clouds were detected in that cell within the modeled environment. I've attached a screenshot of the 0 hour forecast from the RAP on 2025-03-05 11Z run and the Cloud Top Height product from GOES 16 from roughly the same date/time. The NaNf values are transparent in the RR image.

Cheers!
Sean