Evaluating checkpoints

[anderleich]

Hi,
I'm training a model for new english corpora. My question is: during training how can I know which is the best checkpoint according to the evaluation set? I need to know the best model for decoding.

[bfs18]

In my opnion, the validition loss is not a relaible indiactor for the training status since teacher forcing input is used. You can keep out some data, inference on the text and calculate the mel specturm distance between the inferenced mel specturm and ground truth. This indiactor is used in Pre-Alignment Guided Attention for Improving Training Efficiency and Model Stability in End-to-End Speech Synthesis and Location-Relative Attention Mechanisms For Robust Long-Form Speech

[anderleich]

Thanks @bfs18! Do you know some implementation of it in Python?

[bfs18]

You can use this python package https://github.com/MattShannon/mcd

from mcd import dtw, metrics
# ref_data and syn_data are a numpy arrays of shape [n_frames, n_mel]
def _calculate_mcd(ref_data, syn_data):
    try:
        dist, path = dtw.dtw(ref_data, syn_data, metrics.eucCepDist)
        # the denominator is smaller and the avg mcd is larger if repeating or skipping occurs.
        # d = ref_data.shape[0] - np.abs(ref_data.shape[0] - syn_data.shape[0])
        d = ref_data.shape[0]
        return dist / d
    except IndexError:
        return -1.0```

[anderleich]

Thanks, and how do I get ref and syn mel spectrograms from an audio and text pair?

I think this should be included in the repository. I wonder how people gets the best checkpoint with the current code...

[Yeongtae]

Use my branch.
https://github.com/Yeongtae/tacotron2/tree/monitoring_metric
#284

[anderleich]

I've already done the training phase. I wouldn't like to retrain it. Is there a way to just evaluate held out data?

[Yeongtae]

I've already done the training phase. I wouldn't like to retrain it. Is there a way to just evaluate held out data?

Skip the training phase, and evaluate your model on validation set or test set.

[anderleich]

Sorry, but I don't know how

[rafaelvalle]

Closing due to inactivity.