Latest tts

[JRMeyer]

>>> maneeshkyadav
[February 25, 2020, 12:42am]

Hi all, slash
I was working with a TTS version I cloned about a year ago and was very
impressed by the quality out-of-the-box. I was interested in testing out
the latest version with multi-speaker and, after trying to do some
controls on LJSpeech 1.1, the final samples from validation after 1k
epochs sound much worse. I've tried matching the few different config
settings (e.g. learning rate) but i am still getting pretty poor quality
speech on all of LJSpeech over 1000 epochs.

I am trying to play with the attention settings, but has anyone gone
through this? I don't want to try LibriTTS until I am sure that I've got
things working sensibly.

[This is an archived TTS discussion thread from discourse.mozilla.org/t/latest-tts]

[JRMeyer]

>>> maneeshkyadav
[February 26, 2020, 11:02pm]

I should have been more verbose in my description, but I think that the
'gradual learning' was really degrading quality on LJSpeech 1.1.
Setting: slash
'gradual_training': null

seemed to do the trick (not 100% sure, there are a few other parameters
that I changed).

[Archived Post]

[JRMeyer]

>>> erogol
[February 28, 2020, 12:04pm]

can you post tensorboard outputs with and without gradual training?

[Archived Post]

[JRMeyer]

>>> maneeshkyadav
[March 4, 2020, 11:54pm]

Apologies for the delay in responding, I wanted to do a few more runs to
try an establish things but I actually don't think I have a good handle
on what is going on. I successfully got the slash ~1yr old code base
producing pretty reasonable quality speech on about 3 hours of training
data augmented with phonemes (my own hacky code path, not with the way
phonemes are currently implemented). I am struggling to get baseline
performance with the current release on LJ Speech 1.1.

I tried simplifying the config the best I can and I still wind up with
uninterpretable speech after 1000 epochs. In some configurations the
network is producing good output midway through training (similar to
what I got on the much older release) but I can see these same sort of
loss spikes. I've attached my
configconfig.json.zip
(3.2 KB) and a screen shot of loss

AM.filename}[1391×653 60

I must be doing something pretty stupid, can anyone spot it?

[Archived Post]

[JRMeyer]

>>> nmstoker
[March 5, 2020, 12:43am]

Just to check I've understood you correctly, you are struggling to get
good results after 1000 epochs of the current release, ie the TTS repo
as it is in the master branch (or dev?) as it is now? You mention some
separate code with phonemes, but the way I'm reading what you wrote it
seems that's not part of the code you're having trouble with is it?

It's hard to know, but I suspect you'll struggle because you've
switched off the gradual training - you indicated that you did this
because you weren't getting good results, but what you've ended up with
is doing all the training at r=5 which sounds to me like it would be
limiting. Generally the quality of speech improves as you move to lower
values of r (provided that the model managed the transition without
things getting messed up)

How about reinstating the gradual training in your config? I'd be
inclined to put it back in as per the config.json values, however I
suppose you could experiment with not including all the levels, eg just
go down to r=3 and see if that's any better and then add back in the
additional levels and retrain from a checkpoint a little before it
reaches the point where it switches down to the next level.

Is there anything else that might have an impact you haven't mentioned,
such as not using all the LJSpeech dataset, limitations on your GPU
memory, any errors in your logs, older/newer versions of Python modules
installed than the recommended ones or any other unusual setup?

[Archived Post]

[JRMeyer]

>>> nmstoker
[March 5, 2020, 12:53am]

Oh, you've got this in your config:

'use_phonemes': false

That'll make it struggle. Any reason you didn't want that on?

And I suppose you could try turning on use_forward_attn.

And your tensorboard logs only show up to 100k but you previously talked
about going up to 1000 - do you have the longer set?

[Archived Post]

[JRMeyer]

>>> maneeshkyadav
[March 5, 2020, 1:01am]

'Just to check I've understood you correctly, you are struggling to get
good results after 1000 epochs of the current release, ie the TTS repo
as it is in the master branch (or dev?) as it is now?'

Correct, struggling to get good results with the current master branch
on LJ Speech 1.1. The bit about phonemes can be ignored, I only
mentioned to try certify that things had been working nicely with this
codebase without much fuss in my small hands, but something funny is
happening now that I can't quite isolate.

My initial TTS experiments were on dual 1080Tis, my new experiments are
on quad RTX 2080 Tis with newer python/torch (python= 3.6.9,
torch=1.3.0). I don't think there is anything unusual/odd going on in
terms of platform.

I'll put gradual back in and play with r to see how I do.

[Archived Post]

[JRMeyer]

>>> maneeshkyadav
[March 5, 2020, 1:19am]

I am playing with phonemes and I wanted to do some control testing,
that's why I have it off (I do have runs with it on and working, but I
still get have this issue of loss spikes).

The old codebase (with no configurable attention or phonemes) did fine
before, so I'm turning it off as a part of isolating what is going
wrong. I think the horizontal axis is ok? slash 90 steps per epoch slash curl-run-all.sh discourse.mozilla.org html-to-markdown.sh ordered-posts ordered-posts TTS.cdx tts.commands tts-emails.txt TTS.pages tts-telegram.txt TTS.warc.gz 1000
epochs?

[Archived Post]

[JRMeyer]

>>> maneeshkyadav
[March 5, 2020, 2:02am]

For the sake of completeness, some of the graphs I see when I was
training on a 3.3h dataset (with custom phonemeization) with slash ~1 year
old code base (pre configurable attention settings and built-in
phonemeization support). This looks healthier.

PM.filename}[1407×797 85.9

config snippet was pretty simple: slash
'num_freq': 1025, slash
'sample_rate': 22000, slash
'frame_length_ms': 50, slash
'frame_shift_ms': 10.0, slash
'preemphasis': 0.97, slash
'min_level_db': -100, slash
'ref_level_db': 20, slash
'embedding_size': 256, slash
'text_cleaner': 'null_cleaners',

'num_loader_workers': 4,

'epochs': 1000,
'lr': 0.002,
'warmup_steps': 4000,
'lr_decay': 0.5,
'decay_step': 100000,
'batch_size': 32,
'eval_batch_size':-1,
'r': 5,
'wd': 0.0001,

'griffin_lim_iters': 60,
'power': 1.5,

[Archived Post]

[JRMeyer]

>>> maneeshkyadav
[March 6, 2020, 1:44am]

If anyone does have a 'bare bones' config (with the least reliance on
any fancy new features) that they know works fine on LJ Speech with the
current master, I'd love to see it.

[Archived Post]

[JRMeyer]

>>> maneeshkyadav
[March 8, 2020, 5:47am]

Still getting these loss spikes across configurations. slash

PM.filename}[698×580 42.3

I used graves attention model config from: slash

{.site-icon width='32'
GitHub

### mozilla/TTS

forum: https://discourse.mozilla.org/c/tts) - mozilla/TTS

(attached mine just in case) slash
config.json.zip
(3.2 KB)

I pulled the most recent (9b97430) and
now getting an odd NaN issue.

I'm going to sanity check whatever I can (data etc.), but if anyone can
see the issue, I'd love to know.

> Validation slash
> slash | slash > TotalLoss: 6.85920 PostnetLoss: 0.12684 - 0.12684
> DecoderLoss:0.21654 - 0.21654 StopLoss: 6.45301 - 6.45301 AlignScore:
> 0.4279 : 0.4279 slash
> warning: audio amplitude out of range, auto clipped. slash
> slash | slash > Synthesizing test sentences slash
> slash | slash > Training Loss: 0.15940 Validation Loss: 0.12559 slash
> Number of outputs per iteration: 3

> Epoch 358/1000 slash
> slash | slash > Step:16/92 GlobalStep:21075 PostnetLoss:0.32882
> DecoderLoss:0.45036 StopLoss:0.17568 AlignScore:0.1434
> GradNorm:4.90582 GradNormST:2.12395 AvgTextLen:59.9 AvgSpecLen:336.6
> StepTime:1.03 LoaderTime:0.02 LR:0.000100 slash
> slash | slash > Step:41/92 GlobalStep:21100 PostnetLoss:0.28766
> DecoderLoss:0.40615 StopLoss:3.42795 AlignScore:0.1384
> GradNorm:46714.50217 GradNormST:13.98048 AvgTextLen:93.1
> AvgSpecLen:517.1 StepTime:1.37 LoaderTime:0.02 LR:0.000100 slash
> slash | slash > Step:66/92 GlobalStep:21125 PostnetLoss:0.42214
> DecoderLoss:0.55323 StopLoss:0.23890 AlignScore:0.0948
> GradNorm:354.74020 GradNormST:2.37551 AvgTextLen:116.8
> AvgSpecLen:651.6 StepTime:1.67 LoaderTime:0.01 LR:0.000100 slash
> slash | slash > Gradient is INF !! slash
> slash | slash > Gradient is INF !! slash
> slash | slash > Gradient is INF !! slash
> slash | slash > Gradient is INF !! slash
> slash | slash > Step:91/92 GlobalStep:21150 PostnetLoss:nan DecoderLoss:nan
> StopLoss:nan AlignScore:nan GradNorm:nan GradNormST:nan
> AvgTextLen:149.2 AvgSpecLen:776.4 StepTime:0.81 LoaderTime:0.01
> LR:0.000100 slash
> slash [WARNING slash ] NaN or Inf found in input tensor. slash
> slash [WARNING slash ] NaN or Inf found in input tensor. slash
> slash [WARNING slash ] NaN or Inf found in input tensor. slash
> slash [WARNING slash ] NaN or Inf found in input tensor. slash
> slash | slash > EPOCH END -- GlobalStep:21151 AvgPostnetLoss:nan
> AvgDecoderLoss:nan AvgStopLoss:nan AvgAlignScore:nan EpochTime:120.15
> AvgStepTime:1.29 AvgLoaderTime:0.03 slash
> slash [WARNING slash ] NaN or Inf found in input tensor. slash
> slash [WARNING slash ] NaN or Inf found in input tensor. slash
> slash [WARNING slash ] NaN or Inf found in input tensor. slash
> slash [WARNING slash ] NaN or Inf found in input tensor.

> Validation slash
> slash | slash > TotalLoss: nan PostnetLoss: nan - nan DecoderLoss:nan - nan
> StopLoss: nan - nan AlignScore: nan : nan slash
> ! Run is kept in
> /nobackup/myadav1/TTS/output/mky_ljspeech_graves2-bn-March-06-2020_08+21PM-9b97430 slash
> Traceback (most recent call last): slash
> File 'train.py', line 715, in slash
> main(args) slash
> File 'train.py', line 634, in main slash
> epoch) slash
> File 'train.py', line 450, in evaluate slash
> eval_audio = ap.inv_mel_spectrogram(const_spec.T) slash
> File '/home/users/myadav/venvs/sri_tts/TTS/utils/audio.py', line 174,
> in inv_mel_spectrogram slash
> return
> self.apply_inv_preemphasis(self. slash _griffin_lim(S slash curl-run-all.sh discourse.mozilla.org html-to-markdown.sh ordered-posts ordered-posts~ TTS.cdx tts.commands tts-emails.txt TTS.pages tts-telegram.txt TTS.warc.gz slash *self.power)) slash
> File '/home/users/myadav/venvs/sri_tts/TTS/utils/audio.py', line 190,
> in slash _griffin_lim slash
> angles = np.exp(1j slash curl-run-all.sh discourse.mozilla.org html-to-markdown.sh ordered-posts ordered-posts~ TTS.cdx tts.commands tts-emails.txt TTS.pages tts-telegram.txt TTS.warc.gz np.angle(self. slash _stft(y))) slash
> File '/home/users/myadav/venvs/sri_tts/TTS/utils/audio.py', line 199,
> in slash _stft slash
> win_length=self.win_length, slash
> File
> '/home/users/myadav/.virtualenvs/sri_tts/lib/python3.6/site-packages/librosa/core/spectrum.py',
> line 215, in stft slash
> util.valid_audio(y) slash
> File
> '/home/users/myadav/.virtualenvs/sri_tts/lib/python3.6/site-packages/librosa/util/utils.py',
> line 275, in valid_audio slash
> raise ParameterError('Audio buffer is not finite everywhere') slash
> librosa.util.exceptions.ParameterError: Audio buffer is not finite
> everywhere

[Archived Post]

[JRMeyer]

>>> maneeshkyadav
[March 13, 2020, 10:56pm]

Ok, switching to dev and starting with the template config looks like
typical healthy training. Still not quite sure how the stuff above is
happening in master, but it seems to not be there in dev.

[Archived Post]