Restart a training job without the file trainer.pth

[LucaBrugnoli]

Hello,
I would like to know if there is a way to restart a training job without the file trainer.pth. Unfortunately, a fault in the computing infrastructure resulted in the loss of both trainer.pth and last_model.pth, while the other files seems to be fine.

Thank you in advance for any help.

[Linux-cpp-lisp]

Hi @LucaBrugnoli ,

What files do you still have? Your original configuration YAML file? best_model.pth? Any log / wandb run?

[LucaBrugnoli]

I still have the original configuration YAML file, the log and the best_model.pth

[Linux-cpp-lisp]

Do you also have config.yaml from the training directory?

[Linux-cpp-lisp]

Actually, as along as you set seed and dataset_seed originally it shouldn't matter.

[Linux-cpp-lisp]

Basically, with your original YAML config (assuming you set seeds) you can deterministically reconstruct your training and validation sets, so all you have to do is make a copy and add the initialize_from_state model builder to start this new training session from the weights you still have in best_model.pth. (See #235 , #243, #205 for examples.)

You can also check your log/wandb for learning rate drops, and make sure to start at the learning rate you stopped at.

This won't give a perfect restart like trainer.pth would, but should still be enough to get reasonable results, unless I've forgotten something important...

[Linux-cpp-lisp]

Make sure to set a new training runname for this continuiation, which is not a restart! It's just a new training that happens to start off loading the weights from the old one.

[LucaBrugnoli]

Many thanks, with the suggested changes to the config.yaml file I was able to run a new training using the best_model.pth file from the incomplete run as a starting point, with apparently contiguous results.

[Linux-cpp-lisp]

Great! Glad to hear it works.