Small Assimilation Increment in EnKF Analysis Function

[Liu-Jincan]

Issue Description:

Hello everyone,

I've been using DAPPER, and it's indeed a fantastic toolkit for getting started with ensemble assimilation—very cool! However, I've encountered a puzzling issue while testing the EnKF method.

I performed a test on the def EnKF_analysis(E, Eo, hnoise, y, upd_a, stats=None, ko=None) function within the EnKF class. In this test, only the [0] position of the state vector had an observation, and its value was 2.39253506. My concern is that the change (increment) at the [0] position before and after assimilation is very small. Shouldn't it be very close to 2.39253506?

I'm using the DAPPER version from the GitHub repository, downloaded as a zip and installed directly on December 15, 2023.

Input and Output Details:

* HMM.Obs , <class 'dapper.mods.TimeDependentOperator'>, CONSTANT operator sepcified by .Op1:
Operator({
     'M': 1,
     'model': 
         Direct obs. at [0]
         <function partial_Id_Obs.<locals>.model at 0x7f0d3922b820>,
     'noise': 
         GaussRV({
                   'C': <CovMat>
                              M: 1
                           kind: 'diag'
                          trunc: 1.0
                             rk: 1
                           full:
                             [[1.]]
                           diag:
                              [1.],
                  'mu': array([0.]),
                   'M': 1
                 }),
     'linear': 
         Constant matrix
         [[1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
          0.
           0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]
         <function partial_Id_Obs.<locals>.linear at 0x7f0d3922b940>,
     'localizer': 
         <function localization_setup.<locals>.localization_now at
          0x7f0d3922b3a0>
    })




* E , <class 'numpy.ndarray'>, [[1.36059421 0.38880873 0.33458711 0.40318951 0.44490782 0.39981259 0.38518977
  0.34451363 0.38321871 0.37165712 0.40704801 0.40081765 0.40521564 0.42501169
  0.46290415 0.3732177  0.37066489 0.40503846 0.40894069 0.42724891 0.39267102
  0.38230496 0.37354109 0.38538623 0.37660516 0.36202769 0.38257418 0.43654602
  0.40638021 0.4123606  0.41432213 0.40816113 0.33943327 0.35499907 0.33599932
  0.35655364 0.42445722 0.35636009 0.40885757 0.39103804]
 [1.35889366 0.35368369 0.42729124 0.42333683 0.3577139  0.4401907  0.392712
  0.41264987 0.34441315 0.43904735 0.39873112 0.3632221  0.41375271 0.43837536
  0.35880416 0.45092262 0.41942282 0.40305575 0.3400173  0.30828784 0.41010147
  0.43642926 0.48061747 0.43292164 0.32129693 0.33401328 0.41592338 0.39129538
  0.40587457 0.41928544 0.44966683 0.41607096 0.40224151 0.43278745 0.35629633
  0.30944642 0.36032755 0.38422642 0.40665217 0.34108877]]


* Eo , <class 'numpy.ndarray'>, [[1.36059421]
 [1.35889366]]


* hnoise , <class 'dapper.tools.randvars.GaussRV'>, 
  -  C: <CovMat>
      ¦       M: 1
      ¦    kind: 'diag'
      ¦   trunc: 1.0
      ¦      rk: 1
      ¦    full:
      ¦      [[1.]]
      ¦    diag:
      ¦       [1.]
  - mu: [0.]
  -  M: 1


* y , <class 'numpy.ndarray'>, [2.39253506]


* upd_a , <class 'str'>, Sqrt


* E (after analysis), <class 'numpy.ndarray'>, [[1.3605957  0.38883956 0.33450573 0.40317182 0.44498436 0.39977715 0.38518316
  0.34445382 0.38325277 0.37159796 0.40705531 0.40085065 0.40520815 0.42499996
  0.46299553 0.37314949 0.37062209 0.4050402  0.40900119 0.42735334 0.39265572
  0.38225745 0.3734471  0.3853445  0.37665371 0.36205228 0.3825449  0.43658574
  0.40638065 0.41235452 0.41429111 0.40815419 0.33937814 0.35493079 0.33598151
  0.35659499 0.42451351 0.35633563 0.40885951 0.39108189]
 [1.35889516 0.35371455 0.4272098  0.42331913 0.3577905  0.44015523 0.39270539
  0.41259001 0.34444724 0.43898814 0.39873843 0.36325512 0.41374521 0.43836362
  0.35889561 0.45085436 0.41937999 0.40305749 0.34007785 0.30839235 0.41008616
  0.43638171 0.4805234  0.43287988 0.32134551 0.33403789 0.41589408 0.39133513
  0.40587501 0.41927935 0.44963578 0.41606401 0.40218633 0.43271912 0.3562785
  0.3094878  0.36038389 0.38420194 0.40665411 0.34113265]]

Any insights or guidance on what might be causing this behavior would be greatly appreciated!

Thank you!

[Liu-Jincan]

Additionally, I tested the def local_analyses(E, Eo, R, y, state_batches, obs_taperer, mp=map, xN=None, g=0) function within the LETKF class, and observed a similar problem where the change (increment) at the [0] position of the state vector is very small.

Input and Output Details:

* HMM.Obs , <class 'dapper.mods.TimeDependentOperator'>, CONSTANT operator sepcified by .Op1:
Operator({
     'M': 1,
     'model': 
         Direct obs. at [0]
         <function partial_Id_Obs.<locals>.model at 0x7f1601df4820>,
     'noise': 
         GaussRV({
                   'C': <CovMat>
                              M: 1
                           kind: 'diag'
                          trunc: 1.0
                             rk: 1
                           full:
                             [[1.]]
                           diag:
                              [1.],
                  'mu': array([0.]),
                   'M': 1
                 }),
     'linear': 
         Constant matrix
         [[1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
          0.
           0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]
         <function partial_Id_Obs.<locals>.linear at 0x7f1601df4940>,
     'localizer': 
         <function localization_setup.<locals>.localization_now at
          0x7f1601df43a0>
    })


* E , <class 'numpy.ndarray'>, [[1.30717344 0.40801634 0.3750785  0.40237587 0.32207407 0.37713692 0.37890264
  0.36045728 0.39770151 0.40860348 0.37114756 0.38465265 0.40419845 0.42468324
  0.36355158 0.36888291 0.34799194 0.34886717 0.35782782 0.43350419 0.41480796
  0.38730308 0.34134571 0.37368748 0.40393427 0.38280461 0.38663228 0.41004028
  0.38771035 0.40472606 0.38046414 0.33292443 0.41533226 0.39330473 0.39352628
  0.3812622  0.38610828 0.43665475 0.34661976 0.38231704]
 [1.35903075 0.43710811 0.38548605 0.34314745 0.38765361 0.40490957 0.4256461
  0.3826824  0.41377304 0.38294322 0.36850792 0.38818397 0.36029404 0.41073816
  0.35193415 0.44026056 0.40378594 0.35330632 0.40505507 0.41288669 0.40268967
  0.38455357 0.36147986 0.38171022 0.41038392 0.37623194 0.35988951 0.4181357
  0.39048088 0.35513027 0.42701075 0.34487054 0.34499088 0.38950855 0.38999201
  0.35449077 0.35750157 0.36546895 0.38694066 0.39533231]]


* Eo , <class 'numpy.ndarray'>, [[1.30717344]
 [1.35903075]]


* R , <CovMat>
      M: 1
   kind: 'diag'
  trunc: 1.0
     rk: 1
   full:
     [[1.]]
   diag:
      [1.]


* y , [2.39253506]


* state_batches , [array([0]), array([1]), array([2]), array([3]), array([4]), array([5]), array([6]), array([7]), array([8]), array([9]), array([10]), array([11]), array([12]), array([13]), array([14]), array([15]), array([16]), array([17]), array([18]), array([19]), array([20]), array([21]), array([22]), array([23]), array([24]), array([25]), array([26]), array([27]), array([28]), array([29]), array([30]), array([31]), array([32]), array([33]), array([34]), array([35]), array([36]), array([37]), array([38]), array([39])]


* obs_taperer , <function localization_setup.<locals>.localization_now.<locals>.obs_taperer at 0x7f15fbd1f280>


* map , <class 'map'>


* xN , None


* g , 0

* E , after local analysis, <class 'numpy.ndarray'>, [[1.30861345 0.40852841 0.37512053 0.40236885 0.32207407 0.37713692 0.37890264
  0.36045728 0.39770151 0.40860348 0.37114756 0.38465265 0.40419845 0.42468324
  0.36355158 0.36888291 0.34799194 0.34886717 0.35782782 0.43350419 0.41480796
  0.38730308 0.34134571 0.37368748 0.40393427 0.38280461 0.38663228 0.41004028
  0.38771035 0.40472606 0.38046414 0.33292443 0.41533226 0.39330473 0.39352628
  0.3812622  0.38610828 0.43664631 0.34678259 0.38254613]
 [1.36043593 0.4376078  0.38552707 0.3431406  0.38765361 0.40490957 0.4256461
  0.3826824  0.41377304 0.38294322 0.36850792 0.38818397 0.36029404 0.41073816
  0.35193415 0.44026056 0.40378594 0.35330632 0.40505507 0.41288669 0.40268967
  0.38455357 0.36147986 0.38171022 0.41038392 0.37623194 0.35988951 0.4181357
  0.39048088 0.35513027 0.42701075 0.34487054 0.34499088 0.38950855 0.38999201
  0.35449077 0.35750157 0.36546071 0.38709956 0.39555586]]

[patnr]

Hello, and sorry for the late reply. As can be seen from the printouts of the state ensemble E, its spread is very small relative to the observation error variance of R, which is causing the assimilation to mainly rely on the prior and disregard the observation. Whether this is as it should be or is due to insufficient inflation or localisation is not clear from the context. I'm afraid I won't have the capacity to further resolve such questions, but let us know if you figure it out!