Flaky test_adaptive_localization.py::test_that_posterior_generalized_variance_increases_in_cutoff

[berland]

Seen in https://github.com/equinor/ert/actions/runs/6717682519/job/18256070373?pr=6430

FAILED tests/unit_tests/analysis/test_adaptive_localization.py::test_that_posterior_generalized_variance_increases_in_cutoff - AssertionError: assert 0.020660700749797346 <= 0.020546704453971852
 +  where 0.020660700749797346 = <function det at 0x7fd9d25c14b0>(array([[ 0.1579183 , -0.17478591, -0.01605299],\n       [-0.17478591,  0.72895318,  0.02066708],\n       [-0.01605299,  0.02066708,  0.24596514]]))
 +    where <function det at 0x7fd9d25c14b0> = <module 'numpy.linalg' from '/opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/site-packages/numpy/linalg/__init__.py'>.det
 +      where <module 'numpy.linalg' from '/opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/site-packages/numpy/linalg/__init__.py'> = np.linalg
 +  and   0.020546704453971852 = <function det at 0x7fd9d25c14b0>(array([[ 0.1579183 , -0.17478591, -0.01371111],\n       [-0.17478591,  0.72895318,  0.01748329],\n       [-0.01371111,  0.01748329,  0.24416996]]))
 +    where <function det at 0x7fd9d25c14b0> = <module 'numpy.linalg' from '/opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/site-packages/numpy/linalg/__init__.py'>.det
 +      where <module 'numpy.linalg' from '/opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/site-packages/numpy/linalg/__init__.py'> = np.linalg

[kvashchuka]

@berland thanks a lot for reporting!

[berland]

Reoccured in https://github.com/equinor/ert/actions/runs/6770870110/job/18400231951?pr=6444

[kvashchuka]

The two reported fails happened with exactly the same values, which is suspicious:

 AssertionError: assert 0.020660700749797346 <= 0.020546704453971852
 +  where 0.020660700749797346 = <function det at 0x7f0ab97c9e70>(array([[ 0.1579183 , -0.17478591, -0.01605299],\n       [-0.17478591,  0.72895318,  0.02066708],\n       [-0.01605299,  0.02066708,  0.24596514]]))
 +    where <function det at 0x7f0ab97c9e70> = <module 'numpy.linalg' from '/opt/hostedtoolcache/Python/3.9.18/x64/lib/python3.9/site-packages/numpy/linalg/__init__.py'>.det
 +      where <module 'numpy.linalg' from '/opt/hostedtoolcache/Python/3.9.18/x64/lib/python3.9/site-packages/numpy/linalg/__init__.py'> = np.linalg
 +  and   0.020546704453971852 = <function det at 0x7f0ab97c9e70>(array([[ 0.1579183 , -0.17478591, -0.01371111],\n       [-0.17478591,  0.72895318,  0.01748329],\n       [-0.01371111,  0.01748329,  0.24416996]]))
 +    where <function det at 0x7f0ab97c9e70> = <module 'numpy.linalg' from '/opt/hostedtoolcache/Python/3.9.18/x64/lib/python3.9/site-packages/numpy/linalg/__init__.py'>.det
 +      where <module 'numpy.linalg' from '/opt/hostedtoolcache/Python/3.9.18/x64/lib/python3.9/site-packages/numpy/linalg/__init__.py'> = np.linalg

In order to catch what exactly happens, a print message for the cutoff value was added (#6532). Therefore all the reports of this happening are highly appreciated🙌

[oyvindeide]

Just got this now:

FAILED tests/unit_tests/analysis/test_adaptive_localization.py::test_that_posterior_generalized_variance_increases_in_cutoff - AssertionError: Assertion failed with cutoff1=0.5104350837566162 and cutoff2=0.5826545098666468
assert 0.009892851111867187 <= 0.009359477360205124

[oyvindeide]

Another one:

 FAILED tests/unit_tests/analysis/test_adaptive_localization.py::test_that_posterior_generalized_variance_increases_in_cutoff - AssertionError: Assertion failed with cutoff1=0.43820745023216534 and cutoff2=0.502227119260729
assert 0.008917988777692173 <= 0.008912061856440246

[frode-aarstad]

One here too:
FAILED tests/unit_tests/analysis/test_adaptive_localization.py::test_that_posterior_generalized_variance_increases_in_cutoff - AssertionError: Assertion failed with cutoff1=0.5124975879504631 and cutoff2=0.5615161300506735

[kvashchuka]

Thanks a lot for reporting! Should be fixed once I merge #6485 👍