SAMRAI regridding restarts with refinementboxes

[PhilipDeegan]

We are seeing that restart files made with refinement boxes, once restarted, perform regridding such that they are no longer equivalent with the initial simulation which wrote the restart files. The regridding that occurs appears similar to that when both simulations use tagging instead of refinementboxes

LLNL/SAMRAI#199

[nicolasaunai]

test restart fails if the test case with refinement boxes is made so to create at 3 levels hierarchy, as seen in #692, which was thus limited to 2 levels in that mode. This failure occurs because of mismatches between some levels, and possibly related to this regrid issue, as it does not occurs in tagging mode.

test restart log for posterity:

philix@fad:~/git/phare/stage$ python3 tests/simulator/test_restarts.py 
test_mode_conserve dim/interp:1/1
mean advance time = 0.12687911987304687.6344sec
total advance time = 0.6343955993652344
.test_restarts dim/interp:1/1
At :/mkn/r/llnl/samrai/master/samrai/source/SAMRAI/hier/PatchHierarchy.C line :325 message: PHARE_hierarchy:  Using zero `proper_nesting_buffer' values.

At :/mkn/r/llnl/samrai/master/samrai/source/SAMRAI/hier/PatchHierarchy.C line :325 message: PHARE_hierarchy:  Using zero `proper_nesting_buffer' values.

mean advance time = 0.44002017974853513  2.2sec
total advance time = 2.200100898742676
mean advance time = 0.44790172576904297.4479sec
total advance time = 0.44790172576904297
Ftest_restarts dim/interp:1/2
At :/mkn/r/llnl/samrai/master/samrai/source/SAMRAI/hier/PatchHierarchy.C line :325 message: PHARE_hierarchy:  Using zero `proper_nesting_buffer' values.

At :/mkn/r/llnl/samrai/master/samrai/source/SAMRAI/hier/PatchHierarchy.C line :325 message: PHARE_hierarchy:  Using zero `proper_nesting_buffer' values.

mean advance time = 0.5508871078491211 2.754sec
total advance time = 2.7544355392456055
mean advance time = 0.55916476249694820.5592sec
total advance time = 0.5591647624969482
particles.py:Particles::eq failed with: 
Not equal to tolerance rtol=1e-07, atol=1e-12

Mismatched elements: 2 / 2017 (0.0992%)
Max absolute difference: 4.85216589e-10
Max relative difference: 3.73259266e-07
 x: array([ 0.440891, -0.862963, -0.304003, ..., -0.035935,  0.419544,
       -0.382606])
 y: array([ 0.440891, -0.862963, -0.304003, ..., -0.035935,  0.419544,
       -0.382606])
  File "/home/philix/git/phare/stage/pyphare/pyphare/pharesee/particles.py", line 90, in __eq__
    all_assert_sorted(self, that)
  File "/home/philix/git/phare/stage/pyphare/pyphare/pharesee/particles.py", line 201, in all_assert_sorted
    np.testing.assert_allclose(part1.v[idx1,2], part2.v[idx2,2], atol=1e-12)
  File "/opt/py/py/lib/python3.9/site-packages/numpy/testing/_private/utils.py", line 1530, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
  File "/opt/py/py/lib/python3.9/site-packages/numpy/testing/_private/utils.py", line 844, in assert_array_compare
    raise AssertionError(msg)
Ftest_restarts dim/interp:1/3
At :/mkn/r/llnl/samrai/master/samrai/source/SAMRAI/hier/PatchHierarchy.C line :325 message: PHARE_hierarchy:  Using zero `proper_nesting_buffer' values.

At :/mkn/r/llnl/samrai/master/samrai/source/SAMRAI/hier/PatchHierarchy.C line :325 message: PHARE_hierarchy:  Using zero `proper_nesting_buffer' values.

mean advance time = 0.6200533866882324   3.1sec
total advance time = 3.100266933441162
mean advance time = 0.63655638694763180.6366sec
total advance time = 0.6365563869476318
particles.py:Particles::eq failed with: 
Not equal to tolerance rtol=1e-07, atol=1e-12

Mismatched elements: 2 / 1987 (0.101%)
Max absolute difference: 1.17925553e-09
Max relative difference: 1.36845514e-06
 x: array([-0.553397,  0.261233, -0.32579 , ...,  0.333007,  0.783376,
        0.23554 ])
 y: array([-0.553397,  0.261233, -0.32579 , ...,  0.333007,  0.783376,
        0.23554 ])
  File "/home/philix/git/phare/stage/pyphare/pyphare/pharesee/particles.py", line 90, in __eq__
    all_assert_sorted(self, that)
  File "/home/philix/git/phare/stage/pyphare/pyphare/pharesee/particles.py", line 200, in all_assert_sorted
    np.testing.assert_allclose(part1.v[idx1,1], part2.v[idx2,1], atol=1e-12)
  File "/opt/py/py/lib/python3.9/site-packages/numpy/testing/_private/utils.py", line 1530, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
  File "/opt/py/py/lib/python3.9/site-packages/numpy/testing/_private/utils.py", line 844, in assert_array_compare
    raise AssertionError(msg)
F
======================================================================
FAIL: test_restarts_1 (__main__.RestartsTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/py/py/lib/python3.9/site-packages/ddt.py", line 191, in wrapper
    return func(self, *args, **kwargs)
  File "/home/philix/git/phare/stage/tests/simulator/test_restarts.py", line 202, in test_restarts
    checks += check_field(run0.GetB(time), run1.GetB(time))
  File "/home/philix/git/phare/stage/tests/simulator/test_restarts.py", line 184, in check_field
    return  check(qty0, qty1, lambda pd0, pd1: np.testing.assert_equal(pd0.dataset[:], pd1.dataset[:]))
  File "/home/philix/git/phare/stage/tests/simulator/test_restarts.py", line 176, in check
    checker(pd0, pd1)
  File "/home/philix/git/phare/stage/tests/simulator/test_restarts.py", line 184, in <lambda>
    return  check(qty0, qty1, lambda pd0, pd1: np.testing.assert_equal(pd0.dataset[:], pd1.dataset[:]))
  File "/opt/py/py/lib/python3.9/site-packages/numpy/testing/_private/utils.py", line 345, in assert_equal
    return assert_array_equal(actual, desired, err_msg, verbose)
  File "/opt/py/py/lib/python3.9/site-packages/numpy/testing/_private/utils.py", line 934, in assert_array_equal
    assert_array_compare(operator.__eq__, x, y, err_msg=err_msg,
  File "/opt/py/py/lib/python3.9/site-packages/numpy/testing/_private/utils.py", line 844, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Arrays are not equal

Mismatched elements: 5 / 204 (2.45%)
Max absolute difference: 4.92828001e-13
Max relative difference: 4.93326264e-13
 x: array([-1.000041, -0.999738, -1.000983, -0.999347, -1.00021 , -0.999897,
       -0.999443, -1.000754, -0.999965, -0.999697, -1.000085, -1.000374,
       -0.999694, -1.000202, -0.999948, -1.000247, -1.000278, -0.999316,...
 y: array([-1.000041, -0.999738, -1.000983, -0.999347, -1.00021 , -0.999897,
       -0.999443, -1.000754, -0.999965, -0.999697, -1.000085, -1.000374,
       -0.999694, -1.000202, -0.999948, -1.000247, -1.000278, -0.999316,...

======================================================================
FAIL: test_restarts_2 (__main__.RestartsTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/py/py/lib/python3.9/site-packages/ddt.py", line 191, in wrapper
    return func(self, *args, **kwargs)
  File "/home/philix/git/phare/stage/tests/simulator/test_restarts.py", line 201, in test_restarts
    checks += check_particles(run0.GetParticles(time, pops), run1.GetParticles(time, pops))
  File "/home/philix/git/phare/stage/tests/simulator/test_restarts.py", line 181, in check_particles
    return check(qty0, qty1, lambda pd0, pd1: self.assertEqual(pd0.dataset, pd1.dataset))
  File "/home/philix/git/phare/stage/tests/simulator/test_restarts.py", line 176, in check
    checker(pd0, pd1)
  File "/home/philix/git/phare/stage/tests/simulator/test_restarts.py", line 181, in <lambda>
    return check(qty0, qty1, lambda pd0, pd1: self.assertEqual(pd0.dataset, pd1.dataset))
AssertionError: <pyphare.pharesee.particles.Particles object at 0x7f45400c2eb0> != <pyphare.pharesee.particles.Particles object at 0x7f454061f7f0>

======================================================================
FAIL: test_restarts_3 (__main__.RestartsTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/py/py/lib/python3.9/site-packages/ddt.py", line 191, in wrapper
    return func(self, *args, **kwargs)
  File "/home/philix/git/phare/stage/tests/simulator/test_restarts.py", line 201, in test_restarts
    checks += check_particles(run0.GetParticles(time, pops), run1.GetParticles(time, pops))
  File "/home/philix/git/phare/stage/tests/simulator/test_restarts.py", line 181, in check_particles
    return check(qty0, qty1, lambda pd0, pd1: self.assertEqual(pd0.dataset, pd1.dataset))
  File "/home/philix/git/phare/stage/tests/simulator/test_restarts.py", line 176, in check
    checker(pd0, pd1)
  File "/home/philix/git/phare/stage/tests/simulator/test_restarts.py", line 181, in <lambda>
    return check(qty0, qty1, lambda pd0, pd1: self.assertEqual(pd0.dataset, pd1.dataset))
AssertionError: <pyphare.pharesee.particles.Particles object at 0x7f454061fa30> != <pyphare.pharesee.particles.Particles object at 0x7f4540605100>