Skip to content

Commit

Permalink
BF: job_templates: Call tar with --ignore-failed-read
Browse files Browse the repository at this point in the history
After a command completes, it writes to "status.$subjob".  If, after
completing its command, a subjob sees that the status files for all
the other subjobs are in, it claims responsibility for the
post-processing step.  For the datalad-run orchestrators,
post-processing includes calling `find` to get a list of newly added
files and then calling `tar` with these files as input.

Given that the above procedure waits until each command exits, the
hope is that all the output files are created and any temporary files
will have been cleaned up.  But we're hitting into cases [*] where
apparently intermediate files are present for the `find` call but gone
by the time `tar` is called.  This leads to `tar` exiting with a
non-zero status and the post-processing being aborted.

Until someone has a better idea of how to deal with this, instruct
`tar` to exit with zero even if an expected file isn't present.  This
allows post-processing to succeed and the incident will still show up
in the captured stderr.

[*] ReproNim#438 (comment)
  • Loading branch information
kyleam committed Aug 16, 2019
1 parent eb6070c commit 483ece5
Showing 1 changed file with 1 addition and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -29,5 +29,6 @@ mkdir -p "$rootdir/outputs"
tar \
-C "$workdir" \
--files-from "$metadir/togethome" \
--ignore-failed-read \
-cz \
-f "$rootdir/outputs/$jobid.tar.gz"

0 comments on commit 483ece5

Please sign in to comment.