Increase EMA-parameter during training

[Benjamin-Hansson]

Hi, I noticed that the EMA-parameter (called beta in the code, τ in the paper) is not updated during training. In the paper they describe that they increase τ from the start value to 1 during training: "Specifically, we set τ = 1 − (1 − τbase) · (cos(πk/K) + 1)/2 with k the current training step and K the maximum number of training steps."
This makes a huge difference to the validation loss at the end of the training.

[michal-choinski]

@Benjamin-Hansson just to clarify - when did you observe this spiking loss? Was it when the EMA parameter was not updated, or the other way around?