Can you help me understand how to fix the ValueError: L-Moments invalid error?

[tommylees112]

How to craft a useful, minimal bug report

Describe the bug
Using climate data from the Copernicus Data Store I want to create a gridded SPI product. However, I get an error saying that ValueError: L-Moments invalid.

To Reproduce
Steps to reproduce the behaviour:

NOTE: Need to have cdsapi installed
pip install cdsapi

import cdsapi
import xarray as xr

# initalise connection to API
c = cdsapi.Client()

# make api request
api_request = {'area': '6.002/33.501/-5.202/42.283',
 'day': ['01',
         '02',
         '03',
         '04',
         '05',
         '06',
         '07',
         '08',
         '09',
         '10',
         '11',
         '12',
         '13',
         '14',
         '15',
         '16',
         '17',
         '18',
         '19',
         '20',
         '21',
         '22',
         '23',
         '24',
         '25',
         '26',
         '27',
         '28',
         '29',
         '30',
         '31'],
 'format': 'netcdf',
 'month': ['01',
           '02',
           '03',
           '04',
           '05',
           '06',
           '07',
           '08',
           '09',
           '10',
           '11',
           '12'],
 'product_type': 'reanalysis',
 'time': ['00:00'],
 'variable': 'total_precipitation',
 'year': ['1980',
          '1981',
          '1982',
          '1983',
          '1984',
          '1985',
          '1986',
          '1987',
          '1988',
          '1989',
          '1990',
          '1991',
          '1992',
          '1993',
          '1994',
          '1995',
          '1996',
          '1997',
          '1998',
          '1999',
          '2000',
          '2001',
          '2002',
          '2003',
          '2004',
          '2005',
          '2006',
          '2007',
          '2008',
          '2009',
          '2010',
          '2011',
          '2012',
          '2013',
          '2014',
          '2015',
          '2016',
          '2017',
          '2018']}

c.retrieve(
    'reanalysis-era5-pressure-levels-monthly-means',
    api_request,
    'kenya_rainfall_.nc'
)

# convert to format that works with climate_indices
ds = xr.open_dataset(data_dir / 'kenya_rainfall_.nc')
ds = ds.rename({"longitude":'lon',"latitude":'lat'})
attrs = ds.attrs
ds['tp'] = ds.tp * 1000
ds.tp.attrs['units'] = 'mm'
ds.to_netcdf(data_dir/'kenya_rainfall2.nc')

process_climate_indices --index spi --periodicity monthly --netcdf_precip kenya_rainfall2.nc --var_name_precip tp --output_file_base ./kenya_monthly_spi --scales 1 3 6 --calibration_start_year 1980 --calibration_end_year 2017 --multiprocessing all

ERROR TRACE:

2019-05-08  15:54:43 INFO Start time:    2019-05-08 15:54:43.833905
2019-05-08  15:54:44 INFO Computing 1-month SPI/Pearson
2019-05-08  15:54:54 ERROR Failed to complete
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/soge-home/users/chri4118/.conda/envs/crp/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/soge-home/users/chri4118/.conda/envs/crp/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/soge-home/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/climate_indices/__main__.py", line 1331, in _apply_along_axis
    computed_array = np.apply_along_axis(func1d, axis=2, arr=sub_array, parameters=args)
  File "/soge-home/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/numpy/lib/shape_base.py", line 357, in apply_along_axis
    res = asanyarray(func1d(inarr_view[ind0], *args, **kwargs))
  File "/soge-home/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/climate_indices/__main__.py", line 1118, in _spi
    periodicity=parameters["periodicity"],
  File "/soge-home/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/lmoments3/distr.py", line 65, in lmom_fit
    return self._lmom_fit(lmom_ratios)
  File "/soge-home/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/lmoments3/distr.py", line 1277, in _lmom_fit
    raise ValueError("L-Moments invalid")
ValueError: L-Moments invalid
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/soge-home/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/climate_indices/__main__.py", line 1680, in main
    _compute_write_index(kwrgs)
  File "/soge-home/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/climate_indices/__main__.py", line 1004, in _compute_write_index
    args=args,
  File "/soge-home/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/climate_indices/__main__.py", line 1302, in _parallel_process
    pool.map(_apply_along_axis, chunk_params)
  File "/soge-home/users/chri4118/.conda/envs/crp/lib/python3.7/multiprocessing/pool.py", line 290, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/soge-home/users/chri4118/.conda/envs/crp/lib/python3.7/multiprocessing/pool.py", line 683, in get
    raise self._value
ValueError: L-Moments invalid
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/soge-home/users/chri4118/.conda/envs/crp/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/soge-home/users/chri4118/.conda/envs/crp/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/soge-home/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/climate_indices/__main__.py", line 1331, in _apply_along_axis
    computed_array = np.apply_along_axis(func1d, axis=2, arr=sub_array, parameters=args)
  File "/soge-home/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/numpy/lib/shape_base.py", line 357, in apply_along_axis
    res = asanyarray(func1d(inarr_view[ind0], *args, **kwargs))
  File "/soge-home/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/climate_indices/__main__.py", line 1118, in _spi
    periodicity=parameters["periodicity"],
  File "/soge-home/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/lmoments3/distr.py", line 65, in lmom_fit
    return self._lmom_fit(lmom_ratios)
  File "/soge-home/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/lmoments3/distr.py", line 1277, in _lmom_fit
    raise ValueError("L-Moments invalid")
ValueError: L-Moments invalid
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/soge-home/users/chri4118/.conda/envs/crp/bin/process_climate_indices", line 11, in <module>
    sys.exit(main())
  File "/soge-home/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/climate_indices/__main__.py", line 1680, in main
    _compute_write_index(kwrgs)
  File "/soge-home/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/climate_indices/__main__.py", line 1004, in _compute_write_index
    args=args,
  File "/soge-home/users/chri4118/.conda/envs/crp/lib/python3.7/site-packages/climate_indices/__main__.py", line 1302, in _parallel_process
    pool.map(_apply_along_axis, chunk_params)
  File "/soge-home/users/chri4118/.conda/envs/crp/lib/python3.7/multiprocessing/pool.py", line 290, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/soge-home/users/chri4118/.conda/envs/crp/lib/python3.7/multiprocessing/pool.py", line 683, in get
    raise self._value
ValueError: L-Moments invalid

Expected behavior
Expect to produce a gridded SPI product.

Screenshots*
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: Ubuntu
• Version: Ubuntu 16.04.5 LTS

Additional context

[monocongo]

Thanks, @tommylees112. I appreciate the detailed info on how to acquire the input data that causes this issue to appear. I have seen this error before and to be honest I'm not sure yet what causes it. I will look into this, and in the mean time if you have any insights please share. Stay tuned...

[tommylees112]

Thanks for your quick response @monocongo ! Did you manage to solve the issue last time? Is there anything that you think I should try? Thanks again! (love the package btw)

[monocongo]

It looks like this came up another time before in issue #251, I think it was a units error that time but looking at the above your units appear to be fine (mm), and besides that issue was around SPEI (temperature units in Kelvin throwing off the PET calculations, if I remember correctly). Unfortunately this is happening in the bowels of the lmoments3 package that I've used within the Pearson fitting code, and I don't have a solid grasp of that codebase. I had my own code for L-moments before (derived from the same Fortran by Hosking as was lmoments3) but I eventually replaced it with lmoments3, assuming that it was probably more robust (the math is more or less voodoo to me), but I'm really not sure if this is true of not. That project appears to be dormant/unsupported (for example I've had a PR there waiting since last October) and it might behoove me to go back to using my old L-moments code for the Pearson fitting.

[monocongo]

I'm having trouble retrieving the data, @tommylees112 (described here).

Once that's resolved I will look into whether or not there are negative (or otherwise wonky) data values, as this may be the cause of the error (pure conjecture at this point).

[tommylees112]

We had some functions in a notebook file here that worked for me? I don't know if they're any help to you at all?

[monocongo]

Is it possible to post the input data file someplace so I can avoid working out the bugs with the retrieval code, @tommylees112? That will expedite an investigation into this issue, which I hope to resolve pronto. It may be that I just go back to using the L-moments code I had in place before switching over to lmoments3 which hasn't been as trouble-free as I expected...

[tommylees112]

@monocongo https://drive.google.com/file/d/1MQor6rCrhvUTDCh-RFTybMHZ5L1-i9eB/view?usp=sharing

Here's the data! thanks very much
Tommy

<xarray.Dataset>
Dimensions:  (lat: 45, lon: 35, time: 468)
Coordinates:
  * lon      (lon) float32 33.75 34.0 34.25 34.5 34.75 ... 41.5 41.75 42.0 42.25
  * lat      (lat) float32 6.0 5.75 5.5 5.25 5.0 ... -4.0 -4.25 -4.5 -4.75 -5.0
  * time     (time) datetime64[ns] 1980-01-01 1980-02-01 ... 2018-12-01
Data variables:
    tp       (time, lat, lon) float32 ...
Attributes:
    Conventions:  CF-1.6
    history:      2019-05-07 19:44:54 GMT by grib_to_netcdf-2.10.0: /opt/ecmw...

[tommylees112]

@monocongo are there notebooks that show how to interact with your package through python as opposed to via the command line? wanted to incorporate the code more as a python package! Might be a really silly question!

[monocongo]

Check this out @tommylees112, let me know if I can help.

[monocongo]

It turns out that there are negative values in the input dataset, and this was causing the error with the L-Moments calculation. I have now added some checks for negative values that clip negative values to zero, and this appears to have fixed the issue. Also, I've replaced the lmoments3 package with the previous version of L-Moments code I was using before (lmoments3 has started causing other issues so it is high time to excise that package from this code).