Attention weights of graves attention get decreased and reach zero value as iteration goes

[JRMeyer]

>>> byuns9334
[January 16, 2020, 9:58am]

HI, I added GravesAttention class to our custom Tacotron-gst code, and
when we train it, the attention weight value (alpha_t) gets decreased
over iterations and eventually it becomes complete zero vector. Can
anyone help with this issue? or has anyone experienced this issue?

[This is an archived TTS discussion thread from discourse.mozilla.org/t/attention-weights-of-graves-attention-get-decreased-and-reach-zero-value-as-iteration-goes]

[JRMeyer]

>>> erogol
[January 16, 2020, 10:05am]

can you post tensorboard figures?

It is not all the attention goes to zero, only the first step value gets
too big therefore on the figure other steps visually dominated by it.

[Archived Post]

[JRMeyer]

>>> byuns9334
[January 16, 2020, 10:16am]

It looks like this after 100 steps. slash

And It used to look like this using Location attention instead of Graves
attention. slash
![image

[Archived Post]

[JRMeyer]

>>> erogol
[January 16, 2020, 10:18am]

This is just the step 100. Wait more. slash

{.site-icon width='32'

### mozilla/TTS

Deep learning for Text to Speech . Contribute to mozilla/TTS development
by creating an account on GitHub.

[Archived Post]

[JRMeyer]

>>> byuns9334
[January 16, 2020, 10:20am]

okay. I will let you know tomorrow after training 1day. thanks for quick
reply

[Archived Post]

[JRMeyer]

>>> byuns9334
[January 17, 2020, 1:14am]

Hi, the training automatically stopped at iteration 1300, as attention
weights became nan value. At iteration 1300, the attention still looked
like this. slash
Maybe I implemented attention in wrong way. slash
Just want to make sure one thing: Do I need to normalize alpha_t
(attention weights) ?

[Archived Post]

[JRMeyer]

>>> erogol
[January 17, 2020, 8:28am]

Did you implement your own? Because we already have it on master now .

[Archived Post]

[JRMeyer]

>>> byuns9334
[January 17, 2020, 9:09am]

yes, I am already using that and trying to add it to our own custom TTS.

[Archived Post]

[JRMeyer]

>>> erogol
[January 27, 2020, 3:56pm]

I updated the graves attention. It looks much more reliable in my
experiments.

[Archived Post]

[JRMeyer]

>>> byuns9334
[January 27, 2020, 4:26pm]

you mean 'dev' branch of mozilla/TTS repo?

[Archived Post]

[JRMeyer]

>>> byuns9334
[January 28, 2020, 5:33am]

Have you tried GMM attention with multispeaker TTS (Especially
tacotron-gst) ? Does attention alignment work well?

[Archived Post]

[JRMeyer]

>>> erogol
[January 28, 2020, 7:32am]

I've not tried with multi speaker

[Archived Post]