Scheduler is flaky, "Too many realizations failed"

[jonathan-eq]

Describe the bug
The test tests/integration_tests/analysis/test_es_update.py::test_update_multiple_param is very flaky, atleast using scheduler.

FAILED tests/integration_tests/analysis/test_es_update.py::test_update_multiple_param[using_scheduler] - ert.cli.main.ErtCliError: Too many realizations have failed! Number of successful realizations: 9, expected minimal number of successful realizations: 10

(https://github.com/equinor/ert/actions/runs/8420486419/job/23055374718?pr=7521)

Expected behaviour
Not flaky.

Environment

• OS: Mostly macos-latest
• ERT/Komodo release: main
• Python version: Mostly (if not only) 3.12
• Remote/HPC execution involved: Github Actions runner

Additional context
Add any other context about the problem here.

[jonathan-eq]

It is currently marked as skipped due to it blocking PRs.

[eivindjahren]

Seems like this is not isolated to any one test. Realizations are intermittently marked as failing on local queue: https://github.com/equinor/ert/actions/runs/8524062765/job/23347923749

[berland]

This seems to occur at multiple places with the same symptom, "Too many realizations failed", not only when running pytest but also integration tests with poly_example.

When looking at the jobqueue log and in poly_out, all realizations are fine. This points to the problem being the sending of events after all realizations are finished to the ensemble evaluator.

[xjules]

Another one:

FAILED tests/integration_tests/test_cli.py::test_that_the_model_warns_when_active_realizations_less_min_realizations[using_scheduler] - ert.cli.main.ErtCliError: Too many realizations have failed! Number of successful realizations: 4, expected minimal number of successful realizations: 5

https://github.com/equinor/ert/actions/runs/8553593756/job/23437210565?pr=7561

[berland]

See #7569 for a test potentially catching the same issue.