bench:ecal_gaps:allow_failure: true

[wdconinc]

Briefly, what does this PR introduce?

bench:ecal_gaps is failing more often than acceptable, for reasons unrelated to the job. This PR allows the benchmarks to fail.

[veprbl]

Is this meant for specific upstream issue of Today (addressed in #114)? Or just vote of distrust for reliability of this specific benchmark?

[wdconinc]

It seems to be this specific benchmark that most often has issues downstream of container builds:

• https://eicweb.phy.anl.gov/EIC/benchmarks/detector_benchmarks/-/pipelines/108920
• https://eicweb.phy.anl.gov/EIC/benchmarks/detector_benchmarks/-/pipelines/108910

Or just the whole list: https://eicweb.phy.anl.gov/EIC/benchmarks/detector_benchmarks/-/pipelines?page=1&scope=all&status=failed

Usually it's

distributed.comm.core.CommClosedError: in <TCP (closed) Worker->Scheduler local=tcp://[::1]:60324 remote=tcp://localhost:6805>: Stream is closed

[veprbl]

Usually it's

distributed.comm.core.CommClosedError: in <TCP (closed) Worker->Scheduler local=tcp://[::1]:60324 remote=tcp://localhost:6805>: Stream is closed

That's actually not the issue. The actual messages from the client are lost in the noise from the scheduler process. I need to fix that.

The recent ones are a result of a regressions in dask_awkward/dask_histogram. We probably need to pin things in requirements.txt.

There are some spurious ones like

Unable to obtain modification time of file sim_output/backwards_ecal/epic/pi-/100MeV/130to177deg/pi-_100MeV_130to177deg.0000.eicrecon.tree.edm4eic.root although it existed before. It could be that a concurrent process has deleted it while Snakemake was running.

That's inherent to caching being broken by scratch cleanup. We need to either patch snakemake to touch its files or, maybe, we could switch cleanup to use atime.