Final results lpcnet tacotron2 spanish

[JRMeyer]

>>> carlfm01
[October 2, 2019, 4:09am]

Hello, just to share my results.I'm stopping at 47 k steps for tacotron
2:

The gaps seems normal for my data and not affecting the performance.

As reference for others: slash

(931).png.filename}[1409×625 82.1

Final audios: slash
(feature-23 is a mouth twister) slash
47k.zip
(1,0 MB)

Experiment with new LPCNet model:

speech.wav with the old model (male)
female voice, audio generated with real speech features. 600k steps
with 14h of voice. slash
test.zip
(1,1 MB)

It was a surprise for me to see the male voice model generates female
voice.

Now about training speed: slash
My first model took 3h/epoch with 50h of data using a V100. (Trained for
10 epoch)

Now the new female model with 14h of speech took 30min epoch.

> Epoch 1 333333/333333 [==============================] - 1858s 6ms/step - loss: 3.2461
> Epoch 2 9536/333333 [..............................] - ETA: 29:54 - loss: 3.2475

It uses CuDNNGRU so is really fast to train, yes the V100 is pretty
fast but most of the speed comes from the optimized CuDNN.

Of couse I'll share the models, as always.



[No output on generating voice

[This is an archived TTS discussion thread from discourse.mozilla.org/t/final-results-lpcnet-tacotron2-spanish]

[JRMeyer]

>>> erogol
[September 20, 2019, 7:04am]

Results sound quite good. If you like we can link your model on our
model page.

BTW, how do you use LPCnet? Is it LPCNet features you feed to network or
mel_spectrograms? Do you have a branch to check out?

[Archived Post]

[JRMeyer]

>>> carlfm01
[October 3, 2019, 7:32am]

{.avatar

> If you like we can link your model on our model page.

With a good explanation yes, otherwise I think it may confuse people?

{.avatar

> Is it LPCNet features

No, instead of 80d mel we just need 20d which in this case will be the
'cepstral features' the LPC features are computed out of the cepstral
features we feed.

{.avatar

> how do you use LPCnet

Quick summary:

1. We need to train LPCNet using raw pcm 16KHz 16bit mono without
header, all the audios concatenated in a single file, there's
scripts to do so in the fork.
2. We compile dump_data to extract the features out of the audio.
3. Now we train LPCNet with the extracted features.
4. Now that the trained is complete, we use taco=1 to compile
dump_data.
5. Extract the features we need for taco using the compiled dumb_data,
just like step 2.
6. Now with the taco fork we preprocess the dataset.
7. Now we replace the audio folder generated by the preprocess step
with the extracted features.
8. Adapt your feeder if it is broken to match the names of the
features.
9. Train tacotron as usual.
10. To predict we reshape the features from taco save them to a file.
11. With the tool dump_lpcnet and the name of the trained model we
extract the network weight into 2 files 'nnet_data.c and .h'.
12. Move them into src of LPCNet and with do make test_lpcnet taco=1
13. With the compiled test_lpcnet we feed the name of the file predicted
using tacotron and the output name to save the raw pcm.

It looks hard but once you puts your hands on, you will understand.

Forks used for my trainings :

The readme is easy to follow. slash

{.site-icon width='32'

### MlWoo/LPCNet

Efficient neural speech synthesis. Contribute to MlWoo/LPCNet
development by creating an account on GitHub.

And for taco2 : (spanis branch) slash

{.site-icon width='32'

### carlfm01/Tacotron-2

DeepMind slash 's Tacotron-2 Tensorflow implementation. Contribute to
carlfm01/Tacotron-2 development by creating an account on GitHub.

It needs cleanup.

Most important change for taco the hparams and : slash

github.com/carlfm01/Tacotron-2

#### Update feeder.py

by carlfm01 on 10:44AM - 17 Sep 19
UTC

changed 1 files with 4 additions and 2 deletions.

[Archived Post]

[JRMeyer]

>>> carlfm01
[September 20, 2019, 8:30am]

This script will help you to understand:

github.com

#### alokprasad/LPCTron/blob/master/Tacotron-2/training.sh

cd ../dataset
wget https://data.keithito.com/data/speech/LJSpeech-1.1.tar.bz2
tar xvf LJSpeech-1.1.tar.bz2
cd -
python3 preprocess.py --base_dir ../dataset --dataset LJSpeech-1.1
#this generates dataset/training_data folder
#
#├── training_data (1)
#│ ├── audio
#│ ├── linear
#│ └── mels
# └── train.txt

#Now use LPCNET dump_data to convert PCM to NPY and place it in audio
#Training TACATRON2 with lpcnet as Vocoder is not done with audio(dump_data PCM 0 and mels

#wav->s16(pcm)->to npy as meta[0] audio file
#audio-LJ024-0006.npy

#convert to PCM ( header removal)

This file has been truncated. show
original

[Archived Post]

[JRMeyer]

>>> carlfm01
[September 20, 2019, 8:34am]

I would like to mention that we dont need text aligned, I could feed the
100h to train LPCNet, at the time of my training I did not noticed or
think about.

[Archived Post]

[JRMeyer]

>>> erogol
[September 20, 2019, 10:39am]

Ohh I thought you used TTS. I was about the ask for a PR with LPCNet but
I guess, it is harder than expected. But great you shared your progress.
At least it shows that it is possible to use LPCNet. I guess I can try
it after I waste my TODO queue with TTS.

[Archived Post]

[JRMeyer]

>>> carlfm01
[September 20, 2019, 11:53am]

{.avatar

> At least it shows that it is possible to use LPCNet

That's why I ended up using the Tacotron2 repo, honestly reading others
issues I thought I was just wasting my time, now that I know how it
works totally worth the effort to try different experiments including
trying to adapt for TTS.

For my use case I need to deploy on low compute devices, so maybe
TTS+LPCNet is a good combination. The roadblock is that it is Pytorch
and I need to compile with Tensorflow to reuse it from DS and a TTS
model. slash
Right now, I'm able to compile for Windows and use it from C slash # to
compute de features, still not from text due to a missing kernel while
running tacotron model that I'm still searching.

I'm sharing also expecting to see others experiments, hopefully with
TTS.

[Archived Post]

[JRMeyer]

>>> manuel3265
[October 13, 2019, 9:06pm]

. Me podriras hacer el
favor de explicar un poco mas del proceso de utilizar LPCNet + Tacotron
2, te lo agradeceria mucho. Tambien quisiera saber si este repositorio
es la implementacion de estos dos repositorios y si puedo tener los
resultados en español que tu obtuviste con este repositorio
https://github.com/carlfm01/LPCTron

[Archived Post]

[JRMeyer]

>>> carlfm01
[October 2, 2019, 4:04am]

Please try to
keep the thread in English for others can understand, if you want to
continue in Spanish I think
[
ukeador](
helps us moving to common voice Spanish or suggest any other
alternative.

{.avatar

> Me podriras hacer el favor de explicar un poco mas del proceso de
> utilizar LPCNet + Tacotron 2,

Please read comment: slash

{.avatar
(Spanish)](https://discourse.mozilla.org/t/final-results-lpcnet-tacotron2-spanish/45354/3)
drop-close='true'
.box}

> With a good explanation yes, otherwise I think it may confuse people?
> No, instead of 80d mel we just need 20d which in this case will be the
> 'cepstral features' the LPC features are computed out of the cepstral
> features we feed. Quick summary: We need to train LPCNet using raw pcm
> 16KHz 16bit mono without header, all the audios concatenated in a
> single file, there's scripts to do so in the fork. We compile
> dump_data to extract the features out of the audio. Now we train
> LPCNet with the ex...

{.avatar

> Tambien quisiera saber si este repositorio es la implementacion de
> estos dos repositorios y si puedo tener los resultados en español que
> tu obtuviste con este repositorio
> https://github.com/carlfm01/LPCTron

No, the correct forks both pinned at the comment.

{.avatar

> poco mas

Hard to guess what more can I say, or where are you stuck. In order to
help you properly please share the thing that you have tried.

[Archived Post]

[JRMeyer]

>>> carlfm01
[October 2, 2019, 4:29am]

btw, sorry for the delay, here are the trained models : slash

drive.google.com

[.googledocs-onebox-logo
.g-drive-logo}

### tuxvoice.zip

Google Drive file.

[Archived Post]

[JRMeyer]

>>> manuel3265
[October 2, 2019, 4:33am]

for your quick
response. I'll keep the thread in English. Actually, I have data in
Spanish (Latin America), around 100 h (I can't share them, they are
private data. Sorry). But, i want to train a model from zero. At this
moment I started with the first pass of your Quick summary

{.avatar

> We need to train LPCNet using raw pcm 16KHz 16bit mono without header,
> all the audios concatenated in a single file, there's scripts to do so
> in the fork.

I hope to continue with the next steps without any problem. Can I bother
you later if I have a problem? please.

Thanks.

[Archived Post]

[JRMeyer]

>>> carlfm01
[October 2, 2019, 4:42am]

{.avatar

> Can I bother you later if I have a problem

No problem, good luck!

[Archived Post]

[JRMeyer]

>>> manuel3265
[October 2, 2019, 9:55pm]

, I hope you are
well, I want to ask you the following questions, I hope you can answer
them. How long to train LPCNet? how many epochs? What was the loss at
the end of the training?

How do I carry out these steps in tacotron?

{.avatar

> - Now that the trained is complete, we use taco=1 to compile
> dump_data.
> - Extract the features we need for taco using the compiled
> dumb_data, just like step 2.
> - Now with the taco fork we preprocess the dataset.

I would really appreciate if you can answer them. Thank you

[Archived Post]

[JRMeyer]

>>> carlfm01
[October 2, 2019, 10:39pm]

{.avatar

> How long to train LPCNet? how many epochs?

About 15 epochs, but usually I test with the predicted features and if
it is not the desired result I keep training. (or with real features at
early training)

{.avatar

> What was the loss at the end of the training?

This depends on your dataset, for my 50h about 2.4, and for 3h like 3.2

{.avatar

> Now with the taco fork we preprocess the dataset.

Just like LJSpeech dataset, no changes from the usual extraction.

{.avatar

> we use taco=1 to compile dump_data

We need the 20d features to train tacotron, please make sure you always
do 'make clean'

The script for this step looks like:

> mkdir -p /mnt/training_data/audio/ slash
> for i in /mnt/LJSpeech-1.1/pcms/ slash *.s16 slash
> do slash
> /data/home/neoxz/LPCNet/dump_data -test slash $i
> /mnt/training_data/audio/ slash $(basename ' slash $i' slash | cut -d. -f1).npy slash
> done

the preprocess step creates /audios, /mel, and other directory that I
don't remember under /training_data

Noticed how I replace the /audio created by preprocess with /audio
created by the dump_data script.

[Archived Post]

[JRMeyer]

>>> carlfm01
[October 3, 2019, 7:34am]

I was reading the quick summary and noticed that the link to the fork is
broken, is not linking to the correct branch, for the taco2 fork you
need to use spanish branch(changing to your alphabet).

Sorry about that
![:confused:](

[Archived Post]

[JRMeyer]

>>> manuel3265
[October 4, 2019, 12:21am]

DO NOT worry, thank you
very much for the information. I wanted to ask you a question, how do I
do this part on LPCNet ?.

{.avatar

> I test with the predicted features and if it is not the desired result
> I keep training.

I'm sorry but I'm new to this topic.

I executed the following command ./src/test_lpcnet.py spanish / pcms /
test_features.f32 test.s16

Now what should I do with test.s16. To do the test, should I modify
something other than the .h5 file?

after, I transform the test.s16 file with **sox test.s16 -r 16000 -c 1
it is rigth?

Thanks.

[Archived Post]

[JRMeyer]

>>> carlfm01
[October 4, 2019, 12:37am]

./dump_lpcnet.py lpcnet15_384_10_G16_64.h5 (replace with your .h5
file)
./dump_data -test test_input.s16 test_features.f32
previous f32 file)
./test_lpcnet test_features.f32 test.s16 slash
finally convert with sox or ffmpeg the .s16 file to header file
like:

}

```
ffmpeg -f s16le -ar 16k -ac 1 -i test.s16 test-out.wav

[Archived Post]

[JRMeyer]

>>> manuel3265
[October 13, 2019, 9:06pm]

Is it normal to have this
result, considering that train have a loss of 2.5 and going in the
second training epoch?

http://www.mediafire.com/file/eevt7vxat3ni5vk/Spanish_train.zip/file

I attach the files, the original and the output of LPCNet.

Original: audios_archivo-1567550220302976.wav

[Archived Post]

[JRMeyer]

>>> carlfm01
[October 4, 2019, 5:11am]

Please upload the zip to the forums, mediafire pops a lot of spam.

[Archived Post]

[JRMeyer]

>>> manuel3265
[October 4, 2019, 5:27am]

I am a new user and I
cannot attach the zip file. slash
I would like to know what you think about the results.

{.avatar

> How much that your training on? Epoch 2?

yes, the training goes in Epoch 2.

As I am a new user, I can
only reply to the topic 3 times.

Any answer I will make in this answer.

{.avatar

> How many hours?

slash
audio hours? or training?

of audio, it's about 80 hours. From training, about 7 hours.

{.avatar

> Sounds good for epoch 2, train until epoch 10 or so

Thanks, I will do it

[next page
→

[Archived Post]