Multilingual training using fast pitch model.

[RamakrishnaChaitanya]

Hi, i'm planning to using a custom dataset which consists of 'x' number of hindi sentences and 'y' number of english sentences to train the fast pitch model. For this, I have added a custom formatter and a language identifier in the metadata.txt as mentioned here

items.append({"text": text, "audio_file": wav_file, "speaker_name": speaker_name, "language":language, "root_path": root_path})

Can i train the model directly using the procedure as mentioned here?

Basically, my idea is to check the code switching performance of the model during inference (as model should have learned about both the languages, and speakers). Is this directly possible with coqui? If not, where do i need to take care? I'm planning to make the following changes in the configuration i.e.,

use_phonemes=True,
text_cleaner="multilingual_cleaners",
phoneme_language="en-us", #(NOT SURE ABOUT THIS)
phonemizer = "espeak"

I'm not sure on how to edit the values in the character class. Can anyone please help?

"characters": { "characters_class": "", "vocab_dict": null, "pad": "_", "eos": "~", "bos": "^", "blank": "null", "characters": " !%,-.01234568:;?abcdefghijklmnopqrstuvwxyz\u00a0\u0901\u0902\u0903\u0905\u0906\u0907\u0908\u0909\u090a\u090b\u090f\u0910\u0911\u0913\u0914\u0915\u0916\u0917\u0918\u0919\u091a\u091b\u091c\u091d\u091e\u091f\u0920\u0921\u0922\u0923\u0924\u0925\u0926\u0927\u0928\u092a\u092b\u092c\u092d\u092e\u092f\u0930\u0931\u0932\u0933\u0935\u0936\u0937\u0938\u0939\u093c\u093e\u093f\u0940\u0941\u0942\u0943\u0945\u0947\u0948\u0949\u094b\u094c\u094d\u0958\u0959\u095a\u095b\u095c\u095d\u095e\u0960\u200d\u200e\u2013\u2014\u2026",
"punctuations": "!\u00a1'(),-.:;\u00bf? ",
"phonemes": "iy\u0268\u0289\u026fu\u026a\u028f\u028ae\u00f8\u0258\u0259\u0275\u0264o\u025b\u0153\u025c\u025e\u028c\u0254\u00e6\u0250a\u0276\u0251\u0252\u1d7b\u0298\u0253\u01c0\u0257\u01c3\u0284\u01c2\u0260\u01c1\u029bpbtd\u0288\u0256c\u025fk\u0261q\u0262\u0294\u0274\u014b\u0272\u0273n\u0271m\u0299r\u0280\u2c71\u027e\u027d\u0278\u03b2fv\u03b8\u00f0sz\u0283\u0292\u0282\u0290\u00e7\u029dx\u0263\u03c7\u0281\u0127\u0295h\u0266\u026c\u026e\u028b\u0279\u027bj\u0270l\u026d\u028e\u029f\u02c8\u02cc\u02d0\u02d1\u028dw\u0265\u029c\u02a2\u02a1\u0255\u0291\u027a\u0267\u02b2\u025a\u02de\u026b",
"is_unique": true,
"is_sorted": true
},`

[eginhard]

FastPitch (and other models based on the ForwardTTS class like FastSpeech2) don't have support for multilingual TTS in Coqui, you'd need to change the code yourself for that.

[RamakrishnaChaitanya]

Ok, may i know the models that support multiple languages? In the recipes folder, there is a folder named multilingual in which yourtts and vits are mentioned. Are these the only models that have support for multilingual TTS?

[eginhard]

Vits, YourTTS and XTTS