fatal flex scanner internal error--end of buffer missed

[meteoDaniel]

What happened?

I updated my project to the latest eccodes version from 2.30.2 to 2.39.0. Now I am receiving

fatal flex scanner internal error--end of buffer missed

error while parsing ecmwf open data forecast.

What are the steps to reproduce the bug?

I am using python:3.10-slim Docker image as basis and I removed all manually installed eccodes binary.

root@8354140fb24b:/app# python -m eccodes selfcheck
Found: ecCodes v2.39.0.
Library: /usr/local/lib/python3.10/site-packages/eccodes.libs/libeccodes-e85d471c.so
Definitions: /app/eccodes_definitions/dmi_grib_definitions:/app/eccodes_definitions/definitions.edzw-2.30.2-1:/MEMFS/definitions
Samples: /MEMFS/samples
Your system is ready.

As you can see I am having some additional definitions, removing them does not change anything to the fact that using:

import xarray
data = xarray.open_mfdataset(
                local_file_list,
                engine="cfgrib",
                parallel=True,
                concat_dim="valid_time",
                combine="nested",
                backend_kwargs={
                    "indexpath": "",
                    "errors": "ignore",
                    "filter_by_keys": {'typeOfLevel': 'mostUnstableParcel', 'shortName': 'mucape'},                },
            )

leads to the issue named above.

Using cfgrib.open_datasets()works fine.

By changing the ECCODES_DEFINITIONS_PATH I am receiving different Errors.

Version

v2.39.0

Platform (OS and architecture)

python3.10-slim Docker image

Relevant log output

ECCODES ERROR   :  grib_handle_create: Cannot create handle, no definitions found
ECCODES ERROR   :  grib_handle_new_from_file_multi: Cannot create handle
ECCODES ERROR   :  grib_handle_create: Cannot create handle, no definitions found
ECCODES ERROR   :  grib_handle_new_from_file_multi: Cannot create handle
fatal flex scanner internal error--end of buffer missed
ECCODES ERROR   :  Parser: syntax error at line 1 of /MEMFS/definitions/grib2/boot.def
ECCODES ERROR   :  ecCodes Version: 2.39.0

Accompanying data

https://data.ecmwf.int/forecasts/20241129/00z/ifs/0p25/oper/

Organisation

alitiq

[meteoDaniel]

Is anyone here, and knows something about this issue? @alexamici do you know something about this?

[shahramn]

Are you launching multiple threads to decode GRIB messages?
Try turning the "parallel" argument in open_mfdataset to False

[meteoDaniel]

Are you launching multiple threads to decode GRIB messages?

I think this is what happens when calling parallel=True , by default dask.delayed is used

[meteoDaniel]

@shahramn setting parallel to False does not raise the issue. So it works. Does this help to understand the issue?

[shahramn]

The eCodes library needs to be built with thread-safety (mutexes). So if you are building it using cmake, add the following option:
-DENABLE_ECCODES_THREADS=ON

[meteoDaniel]

So I reverted my changes and added the installtion of eccodes again:

RUN wget https://cmake.org/files/v3.20/cmake-3.20.1.tar.gz
RUN tar -xzvf cmake-3.20.1.tar.gz
WORKDIR cmake-3.20.1/
RUN apt-get update && apt-get install --fix-missing -yqq
RUN ./bootstrap
RUN make -j$(nproc) && make install

WORKDIR /

RUN curl https://confluence.ecmwf.int/download/attachments/45757960/eccodes-2.39.0-Source.tar.gz?api=v2 -o /tmp/eccodes-2.39.0-Source.tar.gz
RUN tar -xzf /tmp/eccodes-2.39.0-Source.tar.gz
RUN mkdir build \
    && cd build \
    && cmake /eccodes-2.39.0-Source -DCMAKE_INSTALL_PREFIX=/usr/local/share/eccodes -DENABLE_ECCODES_THREADS=ON -DENABLE_JPG=ON \
    && make install


COPY ./requirements.txt /opt/requirements.txt
RUN pip3 install --no-cache-dir -r /opt/requirements.txt --no-binary eccodes

I am using eccodes=2.39.0 in my requirements.txt here. I added ECCODES_PYTHON_USE_FINDLIBS=1 to my .env.

Now the script terminates without any further information. Same with using cfgrib:

Running python -m eccodes selfcheck

root@6c0426ac3fbb:/app# python -m eccodes selfcheck
/usr/local/lib/python3.10/site-packages/gribapi/__init__.py:23: UserWarning: ecCodes 2.38.0 or higher is recommended. You are running version 2.28.0
  warnings.warn(
Found: ecCodes v2.28.0.
Library: libeccodes.so.0
Definitions: /app/eccodes_definitions/dmi_grib_definitions:/app/eccodes_definitions/definitions.edzw-2.36.0-2:/eccodes-2.39.0-Source/definitions/:/usr/share/eccodes/definitions
Samples: /usr/share/eccodes/samples
Your system is ready.

Where does this version is coming from? What kind of dependency can lead to this downgrade?

[meteoDaniel]

Setting ECCODES_PYTHON_USE_FINDLIBS to 0 will change the output of python -m eccodes selfcheck to:

root@7168fbbebeda:/app# export ECCODES_PYTHON_USE_FINDLIBS=0
root@7168fbbebeda:/app# python -m eccodes selfcheck
Found: ecCodes v2.39.0.
Library: /usr/local/lib/python3.10/site-packages/eccodes.libs/libeccodes-e85d471c.so
Definitions: /app/eccodes_definitions/dmi_grib_definitions:/app/eccodes_definitions/definitions.edzw-2.36.0-2:/eccodes-2.39.0-Source/definitions/:/MEMFS/definitions
Samples: /MEMFS/samples
Your system is ready.

But this does not change anythin that the script just terminates without any excpetion.

[meteoDaniel]

There is something that causes installation of a legcy version of eccodes and I can't figure out where it is coming from

The output currently is:

root@3cabf806d1f1:/app# python -m eccodes selfcheck
/usr/local/lib/python3.10/site-packages/gribapi/__init__.py:23: UserWarning: ecCodes 2.38.0 or higher is recommended. You are running version 2.28.0
  warnings.warn(
Found: ecCodes v2.28.0.
Library: libeccodes.so.0
Definitions: /eccodes-2.39.0-Source/definitions:/app/eccodes_definitions/dmi_grib_definitions:/app/eccodes_definitions/definitions.edzw-2.36.0-2:/usr/share/eccodes/definitions
Samples: /usr/share/eccodes/samples
Your system is ready

Here is my Dockerfile:

FROM python:3.10-slim

ENV DEBIAN_FRONTEND=noninteractive
ENV TERM=linux

RUN set -ex \
    && apt update \
    && apt install -y apt-transport-https curl


WORKDIR /tmp
RUN set -ex \
    && buildDeps=' \
        libnetcdf-dev \
        libhdf5-dev \
        libbz2-dev \
    ' \
    && apt-get update -yqq \
    && apt-get install -yqq --no-install-recommends \
        $buildDeps \
        cdo \
        wget \
        bzip2 \
        liblcms2-dev \
        libtiff-dev \
        libpng-dev \
        libz-dev \
        libaec-dev \
        gfortran \
        protobuf-compiler \
        build-essential \
        libssl-dev \
        libopenjp2-7-dev \
        libopenjp2-tools \
    && apt-get remove -y $buildDeps \
    && apt-get autoremove -y \
    && apt-get update


RUN wget https://cmake.org/files/v3.20/cmake-3.20.1.tar.gz
RUN tar -xzvf cmake-3.20.1.tar.gz
WORKDIR cmake-3.20.1/
RUN apt-get update && apt-get install --fix-missing -yqq
RUN ./bootstrap
RUN make -j$(nproc) && make install

WORKDIR /

RUN curl https://confluence.ecmwf.int/download/attachments/45757960/eccodes-2.39.0-Source.tar.gz?api=v2 -o /tmp/eccodes-2.39.0-Source.tar.gz
RUN tar -xzf /tmp/eccodes-2.39.0-Source.tar.gz
RUN mkdir build \
    && cd build \
    && cmake /eccodes-2.39.0-Source -DCMAKE_INSTALL_PREFIX=/usr/local/share/eccodes -DENABLE_ECCODES_THREADS=ON -DENABLE_JPG=ON \
    && make install


COPY ./requirements.txt /opt/requirements.txt
RUN pip3 install --no-dependencies --no-cache-dir -r /opt/requirements.txt
RUN curl https://files.pythonhosted.org/packages/c2/a4/fb4c7e2f7df631ff2cfdf345f6978452c00bdc24b0c9eff33081d6d533f4/eccodes-2.39.0-py3-none-any.whl -o /tmp/eccodes-2.39.0-py3-none-any.whl
RUN pip3 install --no-dependencies --no-cache-dir /tmp/eccodes-2.39.0-py3-none-any.whl
RUN curl https://files.pythonhosted.org/packages/62/c6/7e8a2b07d2404a79b2cea577962f88e02662ba2b85e073b6e5eed8081878/cfgrib-0.9.14.1-py3-none-any.whl -o /tmp/cfgrib-0.9.14.1-py3-none-any.whl
RUN pip3 install --no-dependencies --no-cache-dir /tmp/cfgrib-0.9.14.1-py3-none-any.whl


ENV PYTHONPATH="/app:/app"
#COPY entrypoint.sh /
#COPY .env .env
WORKDIR /app
#COPY entrypoint.sh /
#COPY .env /app/.env
COPY . .
#ENTRYPOINT ["/entrypoint.sh"]

And here is my requirements.txt:

numpy==1.25.2
scipy==1.10.1
pandas==2.1.0
SQLAlchemy==2.0.20
psycopg2-binary==2.9.7
PyYAML==6.0.1
toolz==0.11.2
cachetools==4.1
aiofiles==23.1.0
ipython==8.11.0
plotly==5.16.1
matplotlib==3.3.4
Pillow==8.1.1
pykml==0.2.0
requests==2.31.0
pytest==7.3.1
pytest-cov==4.0.0
mock==0.7.2
pyproj==3.6.0
fire==0.3.1
aiohttp==3.9.2
parse==1.19.0
ecmwf-api-client==1.6.3
jinja2==3.1.4
shapely==2.0.1
geopandas==0.13.2
nest_asyncio==1.4.1
uvloop==0.15.2
netCDF4==1.6.4
h5py==3.9.0
h5netcdf==1.2.0
pygrib==2.1.4
aiobotocore[awscli,boto3]==2.13.1
s3fs==2024.6.1
zarr==2.17.2
xarray==2024.11.0
dask==2024.11.2
distributed==2024.11.2
dask-cloudprovider==2022.10.0
MarkupSafe==2.1.5
cdsapi==0.7.0
asyncssh==2.13.2
aioftp==0.21.4
rechunker==0.5.2
# eccodes and cfgrib dependencies
attrs==19.2.0
atomicwrites==1.3.0
apipkg==1.5
six==1.12.0
idna==3.7  # via requests
alabaster==0.7.12
babel==2.9.1
certifi==2024.7.4
charset-normalizer==3.3.2
docutils==0.14
packaging==19.0
pygments==2.15.0
pyparsing==2.3.1
pytz==2018.9
six==1.12.0
snowballstemmer==1.2.1
sphinx==1.8.5
sphinxcontrib-websupport==1.1.0
urllib3==1.26.19
cffi==1.17.1
findlibs==0.0.5
pycparser==2.22

Hopefully you can figure out where the old installation is coming from.