Snakemake retrains even when not needed

[AndyMcAliley]

If the same training run is triggered twice using Snakemake, the model gets trained again. Snakemake should report that there is nothing to be done.

For example:

$ snakemake -c1 3_train/out/model_prep/t1/a_weights.pt
# training finishes as expected
$ snakemake -c1 3_train/out/model_prep/t1/a_weights.pt
# training runs again - unexpected!

# show the reason for running
$ snakemake -n -r 3_train/out/model_prep/t1/a_weights.pt
Building DAG of jobs...
Job stats:
job            count    min threads    max threads
-----------  -------  -------------  -------------
train_model        1              1              1
total              1              1              1


[Thu Jul 28 08:41:16 2022]
rule train_model:
    input: 2_process/out/model_prep/train.npz, 2_process/out/model_prep/valid.npz, 3_train/out/model_prep/t1/t1_process.yaml, 3_train/out/model_prep/t1/t1_train.yaml
    output: 3_train/out/model_prep/t1/a_weights.pt, 3_train/out/model_prep/t1/a_metadata.npz
    jobid: 0
    reason: Input files updated by another job: 2_process/out/model_prep/valid.npz, 2_process/out/model_prep/train.npz
    wildcards: data_source=model_prep, run_id=t1, model_id=a
    resources: tmpdir=/var/folders/4f/_lj710zn3sq0ds_860n_ytv8js_f4w/T

Job stats:
job            count    min threads    max threads
-----------  -------  -------------  -------------
train_model        1              1              1
total              1              1              1


This was a dry-run (flag -n). The order of jobs does not reflect the order of execution.

I'm not sure why this happens. Here's one possibility. I'm testing on a subset of sequences to speed up development. To do this, I change upstream inputs. Despite my efforts, the subset isn't always the same. TBD if this happens when the full set of sequences is used.

NOTE: This behavior only occurs for snakemake 7.8 and later. Snakemake rules for triggering reruns changed then. See here and here. To replicate the older behavior, use snakemake --rerun-triggers mtime.

[AndyMcAliley]

Possible explanation for constant remake issue: Issue commment - Announcement: Change in rerun behavior in Snakemake 7.8 · Issue #1694 · snakemake/snakemake. Basically, deleting the .snakemake directory and re-running the pipeline may fix it, but other folks are experiencing the same problem even after trying this fix. It takes several hours to run the full pipeline from data processing through model training, so I'll try deleting .snakemake the next time I train a model.