Add post/pre experiment simulation hooks

[yngve-sk]

Issue
Resolves #7644

[oyvindeide]

Looks good, but think we need to adjust the timing a bit. Also it seems to be missing for some experiment types. Though we should not run the PRE_EXPERIMENT hooks for restart runs and evaluate_experiment´ and none of them for manual_update. Also need to add documentation for when these are run, and for which run model. There is another test: test_model_hook_order` which should proably assert this.

[oyvindeide]

Looks good, think the test can be improved a bit though

[oyvindeide]

Could parameterize PRE and POST so we dont check two different (though related) things in the same test. Think it will shorten the test quite a bit also. Could also parameterize the output if you want different for the two hooks. Should also be possible to assert on the stdout if you just print in the scripts: https://docs.pytest.org/en/stable/how-to/capture-stdout-stderr.html#accessing-captured-output-from-a-test-function

[oyvindeide]

For example:

Suggested change

	@pytest.mark.usefixtures("copy_poly_case")
	def test_that_pre_post_experiment_hook_works(
	monkeypatch,
	):
	monkeypatch.setattr(_ert.threading, "_can_raise", False)
	# The executable
	with open("hello_post_exp.sh", "w", encoding="utf-8") as f:
	f.write(
	dedent("""#!/bin/bash
	echo "just sending regards" > from_post_experiment.txt
	""")
	)
	os.chmod("hello_post_exp.sh", 0o755)
	# The workflow job
	with open("SAY_HELLO_POST_EXP", "w", encoding="utf-8") as s:
	s.write("""
	INTERNAL False
	EXECUTABLE hello_post_exp.sh
	""")
	# The workflow
	with open("SAY_HELLO_POST_EXP.wf", "w", encoding="utf-8") as s:
	s.write("""dump_final_ensemble_id""")
	# The executable
	with open("hello_pre_exp.sh", "w", encoding="utf-8") as f:
	f.write(
	dedent("""#!/bin/bash
	echo "first" > from_pre_experiment.txt
	""")
	)
	os.chmod("hello_pre_exp.sh", 0o755)
	# The workflow job
	with open("SAY_HELLO_PRE_EXP", "w", encoding="utf-8") as s:
	s.write("""
	INTERNAL False
	EXECUTABLE hello_pre_exp.sh
	""")
	# The workflow
	with open("SAY_HELLO_PRE_EXP.wf", "w", encoding="utf-8") as s:
	s.write("""dump_first_ensemble_id""")
	with open("poly.ert", mode="a", encoding="utf-8") as fh:
	fh.write(
	dedent(
	"""
	NUM_REALIZATIONS 2
	LOAD_WORKFLOW_JOB SAY_HELLO_POST_EXP dump_final_ensemble_id
	LOAD_WORKFLOW SAY_HELLO_POST_EXP.wf POST_EXPERIMENT_DUMP
	HOOK_WORKFLOW POST_EXPERIMENT_DUMP POST_EXPERIMENT
	LOAD_WORKFLOW_JOB SAY_HELLO_PRE_EXP dump_first_ensemble_id
	LOAD_WORKFLOW SAY_HELLO_PRE_EXP.wf PRE_EXPERIMENT_DUMP
	HOOK_WORKFLOW PRE_EXPERIMENT_DUMP PRE_EXPERIMENT
	"""
	)
	)
	run_cli(ITERATIVE_ENSEMBLE_SMOOTHER_MODE, "--disable-monitor", "poly.ert")
	assert (Path(os.getcwd()) / "from_pre_experiment.txt").read_text(
	"utf-8"
	) == "first\n"
	assert (Path(os.getcwd()) / "from_post_experiment.txt").read_text(
	"utf-8"
	) == "just sending regards\n"
	@pytest.mark.parametrize("hook", ["PRE", "POST"])
	def test_that_pre_post_experiment_hook_works(
	monkeypatch, tmp_path, hook, capsys
	):
	monkeypatch.chdir(tmp_path)
	monkeypatch.setattr(_ert.threading, "_can_raise", False)
	# The executable
	with open("hello.sh", "w", encoding="utf-8") as f:
	f.write(
	dedent("""#!/bin/bash
	echo "just sending regards"
	""")
	)
	os.chmod("hello.sh", 0o755)
	# The workflow job
	with open("SAY_HELLO", "w", encoding="utf-8") as s:
	s.write("""
	INTERNAL False
	EXECUTABLE hello.sh
	""")
	# The workflow
	with open("SAY_HELLO_CONFIG", "w", encoding="utf-8") as s:
	s.write("""SAY_HELLO""")
	with open("poly.ert", mode="a", encoding="utf-8") as fh:
	fh.write(
	dedent(
	f"""
	NUM_REALIZATIONS 2
	LOAD_WORKFLOW_JOB SAY_HELLO
	LOAD_WORKFLOW SAY_HELLO_CONFIG
	HOOK_WORKFLOW SAY_HELLO_CONFIG {hook.upper()}_EXPERIMENT
	"""
	)
	)
	run_cli(ENSEMBLE_EXPERIMENT_MODE, "--disable-monitor", "poly.ert")
	captured = capsys.readouterr()
	assert "just sending regards" in captured.out

could also look at: test_model_hook_order, so it can be tested for all relevant experiments, and not just ies

[yngve-sk]

I think the runtime of the test will be quite long if we run it for all modes, we also need observations to run anything beyond ensemble experiment so I think the copy poly (or creating observations, having params and running a forward model + responses is needed). Running with parametrized PRE/POST with ensemble_experiment (and copy poly) took 54s for me locally, so I think maybe keeping it verbose and for ES only might be the way. We test that the hooks are invoked in test_model_hook_order. Running for all modes with capsys without parametrizing PRE/POST takes 47s for me locally.

running the test below took ~5min locally

@pytest.mark.usefixtures("copy_poly_case")
@pytest.mark.parametrize("hook", ["PRE", "POST"])
def test_that_pre_post_experiment_hook_works(monkeypatch, hook, capsys):
    monkeypatch.setattr(_ert.threading, "_can_raise", False)
    # The executable
    with open("hello.sh", "w", encoding="utf-8") as f:
        f.write(
            dedent("""#!/bin/bash
                echo "just sending regards"
        """)
        )
    os.chmod("hello.sh", 0o755)
    # The workflow job
    with open("SAY_HELLO", "w", encoding="utf-8") as s:
        s.write("""
               INTERNAL False
               EXECUTABLE hello.sh
           """)
    # The workflow
    with open("SAY_HELLO_CONFIG", "w", encoding="utf-8") as s:
        s.write("""SAY_HELLO""")

    with open("poly.ert", mode="a", encoding="utf-8") as fh:
        fh.write(
            dedent(
                f"""
                    LOAD_WORKFLOW_JOB SAY_HELLO
                    LOAD_WORKFLOW SAY_HELLO_CONFIG
                    HOOK_WORKFLOW SAY_HELLO_CONFIG {hook.upper()}_EXPERIMENT
                """
            )
        )

    for mode in [
        ENSEMBLE_EXPERIMENT_MODE,
        ENSEMBLE_SMOOTHER_MODE,
        ITERATIVE_ENSEMBLE_SMOOTHER_MODE,
        ES_MDA_MODE,
    ]:
        run_cli(mode, "--disable-monitor", "poly.ert")
        captured = capsys.readouterr()
        assert "just sending regards" in captured.out

[codspeed-hq]

CodSpeed Performance Report

Merging #8993 will not alter performance

_{Comparing yngve-sk:add-post-experiment-hook (9637946) with main (864c1e8)}

Summary

✅ 24 untouched benchmarks

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 91.72%. Comparing base (864c1e8) to head (9637946).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8993      +/-   ##
==========================================
- Coverage   91.73%   91.72%   -0.01%     
==========================================
  Files         430      430              
  Lines       26639    26649      +10     
==========================================
+ Hits        24436    24443       +7     
- Misses       2203     2206       +3     

Flag	Coverage Δ
cli-tests	39.71% <100.00%> (+0.02%)	⬆️
everest-models-test	34.18% <38.46%> (-0.01%)	⬇️
gui-tests	71.84% <84.61%> (+<0.01%)	⬆️
integration-test	38.29% <38.46%> (-0.03%)	⬇️
performance-tests	51.69% <69.23%> (+<0.01%)	⬆️
test	39.61% <38.46%> (-0.03%)	⬇️
unit-tests	74.00% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.