-
Notifications
You must be signed in to change notification settings - Fork 16
FAQ
It is very common, that NWP (numerical weather prediction) files are updated while the model is running. It is often desirable to be able to read the file already when the model is still running, e.g. to see the first few ours of weather prediction.
Fimex does not use file-locks for reading: Graphical interfaces like
diana which read via Fimex might keep the file
open as long as it does not change, therefor locking the file for too
long to allow writers to update the files. Writing additional data to
an existing file is usually not a problem, in particularly when the
reader re-opens the file when it has updated the time-stamp mtime.
Fimex does not implement file-locking for writers generally, though in particularly with the fillWriter interface this situation might change, in particular for netcdf-files
If the original variable e.g. t2m has units-attribute Celsius and
you want to have a new netcdf-file with unit Kelvin, you should use
the --output.config=cdmWriterConfig.xml and define there the new
units for the that variable and make sure you defined the datatype of
that variable, too.
If the input-file has the wrong units, e.g. Celsius while the data in fact is Kelvin, you should use the input.config=... and re-define the units-attribute. The data-values won't be changed.
I have problems with a felt-file with a variable of type
'short'. fimex abborts with change in scaling-factor for parameter: 31,2,1000 consider using float or double datatpye. What should I do?
-
While the netcdf-format allows for only 1 scaling-factor per variable, the felt-format can have a scaling-factor per 2d-slice per variable (that are level*time scaling-factors). You should change the
typeof the variable and its_FillValueattribute fromshorttofloatin the filefelt2nc_variables.xml. -
If you want to save disk-space, and you know that all data of the variable can be subscribed sufficiently with a packed short data-array, use the ncml-configuration as follows:
<?xml version="1.0" encoding="UTF-8"?> <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2 ncml-2.2.xsd "> <!-- change the type of a variable, repack --> <variable name="precipitation_amount" type="short"> <attribute name="_FillValue" value="-32767" type="short" /> <attribute name="scale_factor" value="0.1" type="float" /> <attribute name="add_offset" value="0" type="double" /> </variable> </netcdf>
In this example, the precipitation will be described with an accuracy
of 0.1mm and up to 3276.7mm. The ncml-configfile will be applied right
before writing with the --ncml.config=....ncml fimex-option.
Vectors are autorotated if one of the following applies:
- the
<spatial_vector direction="x" counterpart="y_wind" />property is set for both vectors, this can be done with ncml, or the different reader-configurations - the vector has a known CF-standard-name, e.g.
- x_wind, y_wind
- _x_velocity,_y_velocity
- _x_transport,_y_transport
- _x_displacement,_y_displacement
- eastward, northward
- the complete list can be found in the source code of
enhanceVectorPropertieshere
When rotating vectors for spatial interpolations, fimex does not change the scale_factor, add_offset or unit of the vector-components, but requires, that those are equal for both x and y component of the vector. This happens in particular when retrieving wind-data from ECMWF in netcdf, e.g. for ERA data.
A solution is to configure the vector to have the same
scale/add_offset, e.g. with ncml (--input.config):
<variable orgName="Uvind" name="x_wind" type="float">
<attribute name="scale_factor" value="1" type="float" />
<attribute name="add_offset" value="0" type="float" />
<remove name="_FillValue" type="attribute" />
<attribute name="standard_name" value="x_wind" type="String" />
</variable>
<variable orgName="Vvind" name="y_wind" type="float" >
<attribute name="scale_factor" value="1" type="float" />
<attribute name="add_offset" value="0" type="float" />
<remove name="_FillValue" type="attribute" />
<attribute name="standard_name" value="y_wind" type="String" />
</variable>
The config-file for felt-files must at least define 1 existing parameter in the corresponding felt-file. Update your config-file (e.g. felt2nc_variables.xml) with the output from the felt-index, which can be read by felt-tools like rfinh.
This error-message is unfortuneatly very misleading. In many cases,
the _FillValue type does not correspond to the variables
type. Please check your config-file. Unless really required, it's
recommended to remove the _FillValue.
Since fimex 0.40.1, it is possible to give multiple optional file-sources:
fimex --input.file=/opdata/lameps/mle/eceps_201211260000_ens00.mars \
--input.type=grb --input.config=/home/heikok/Programme/MetSis/Fimex/share/etc/cdmGribReaderConfig.xml \
--input.optional=/opdata/lameps/mle/eceps_201211260000_ens01.mars \
--input.optional=/opdata/lameps/mle/eceps_201211260000_ens02.mars \
--output.file=eceps.nc4
Example:
fimex --input.file=http://thredds.met.no/thredds/dodsC/topaz/dataset-topaz3-arc-myocean-be \
--input.type=netcdf \
--output.type=netcdf \
--output.file=test.nc \
\
--extract.selectVariables=time \
--extract.selectVariables=x \
--extract.selectVariables=y \
--extract.selectVariables=depth \
--extract.selectVariables=longitude \
--extract.selectVariables=latitude \
--extract.selectVariables=temperature \
\
--extract.reduceTime.start=2010-10-05 \
--extract.reduceTime.end=2010-10-07 \
\
--extract.reduceToBoundingBox.south 55.0 \
--extract.reduceToBoundingBox.north 80.0 \
--extract.reduceToBoundingBox.east 20.0 \
--extract.reduceToBoundingBox.west -5.0
OpENDAP servers are often limited on how much data can be read at per request, e.g. thredds allows only 500MB binary data / 50MB ascii data per request before failing with an error like
syntax error, unexpected WORD_STRING, expecting WORD_WORD
context: Error { code = 403; message = "Request too big=19011.0 Mbytes, max=500.0"^;};
terminate called after throwing an instance of 'MetNoFimex::CDMException'
what(): CDMException: NetCDF: Malformed or inaccessible DAP DATADDS
Aborted
Fimex tries to subdivide requests into one unlimited slice per
variable. Unfortunately, OpENDAP often does not know about 'unlimited'
dimension and one has to add that manually to fimex, e.g. for an
unlimited time dimension: --input.config=unlimitedTime.ncml with
contents like:
<?xml version="1.0" encoding="UTF-8"?>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2 http://www.unidata.ucar.edu/schemas/netcdf/ncml-2.2.xsd">
<dimension name="time" isUnlimited="true" />
</netcdf>
In theory, it should be possible to specify username and password
- in
$HOME/.dodsrcand$HOME/.netrc(see below for an example) - as part of the URL for example
https://XXXX:[email protected]/thredds/dodsC/....nc
Example $HOME/.netrc:
machine thredds.met.no
login XXXX
password YYYY
Example $HOME/.dodsrc (replace YOURHOME with the value og $HOME):
HTTP.NETRC=YOURHOME/.netrc
Unfortunately, this does not always seem to work as expected.
Known status of OpenDAP authentication support (applies to fimex and ncdump):
- Specifying the password in the URL does not seem to work when the password contains a
/, at least for Ubuntu 18.04. - For Ubuntu 16.04 (xenial, EOL), both URL and netrc seem to work.
- For Ubuntu 18.04 (bionic), username and password cannot be read from netrc, but can be specified in the URL.
- For Ubuntu 20.04 (focal), username and password can only be read from netrc, but cannot be specified in the URL.