Changing the path when resuming from a checkpoint that differs from the one in train.json.

[azrahello]

First of all, I want to wish you a Merry Christmas and thank you for this wonderful Christmas gift! Amazing work! I noticed that when resuming training from a checkpoint, the continuation uses the directory specified in the train.json file, but it’s not necessarily the correct one, as the directory might actually be directory_+_timestamp.

[filipstrand]

Thank you, and happy Christmas to you too :)

Thanks for pointing this out, it should be fixed.

Another small bug I noted is the duration in the checkpoint.json can sometimes be wrong when starting and resuming training with some hours in-between runs, for example:

1. Train for 1h
2. Pause for 1h
3. Train for 1h more

this will report a 3h duration, when it should be 2h.

This happens since we only log the start time, but it should probably work more like a stopwatch since training can be assumed to start and stop at arbitrary times.

[azrahello]

Yes, I noticed it too. I started a training session in the evening and resumed it the next day, and the duration was indeed gross. I didn’t perceive it as an “issue.” In my fork, I shared the file plotter.py. If you feel like it, take a look—it’s my first approach to Python. I relied heavily on AIs for help, but I find it “fun.”