| Tests |     |
|---|---|
| Release |    |
| Citation |   |
-
The phonemizer allows simple phonemization of words and texts in many languages.
-
Provides both the
phonemizecommand-line tool and the Python functionphonemizer.phonemize. See function documentation. -
It is based on four backends: espeak, espeak-mbrola, festival and segments. The backends have different properties and capabilities resumed in table below. The backend choice is let to the user.
-
espeak-ng is a Text-to-Speech software supporting a lot of languages and IPA (International Phonetic Alphabet) output.
-
espeak-ng-mbrola uses the SAMPA phonetic alphabet instead of IPA but does not preserve word boundaries.
-
festival is another Tex-to-Speech engine. Its phonemizer backend currently supports only American English. It uses a custom phoneset, but it allows tokenization at the syllable level.
-
segments is a Unicode tokenizer that build a phonemization from a grapheme to phoneme mapping provided as a file by the user.
espeak espeak-mbrola festival segments phone set IPA SAMPA custom user defined supported languages 100+ 35 US English user defined processing speed fast slow very slow fast phone tokens ✔️ ✔️ ✔️ ✔️ syllable tokens ❌ ❌ ✔️ ❌ word tokens ✔️ ❌ ✔️ ✔️ punctuation preservation ✔️ ❌ ✔️ ✔️ stressed phones ✔️ ❌ ❌ ❌ tie ✔️ ❌ ❌ ❌ -
You need python>=3.6. If you really need to use python2, use the phonemizer-1.0 release.
You need to install
festival,
espeak-ng
and/or mbrola in order to use the
corresponding phonemizer backends. Follow instructions for your system below.
on Debian/Unbuntu
To install dependencies, simply run sudo apt-get install festival espeak-ng mbrola.
When using the espeak-mbrola backend, additional mbrola voices must be
installed (see
here). List
the installable voices with apt search mbrola.
on CentOS/Fedora
To install dependencies, simply run sudo yum install festival espeak-ng.
When using the espeak-mbrola backend, the mbrola binary and additional mbrola voices must be installed (see here).
on MacOS
espeak is available on brew at version 1.48: brew install espeak. If you
want a more recent version you have to compile it from
sources.
To install festival, mbrola and additional mbrola voices, use the
script provided here.
on Windows
Install espeak-ng with the .msi Windows installer provided with espeak
releases. festival must be
compiled from sources (see
here and
here). mbrola is
not available for Windows.
-
The simplest way is using pip:
pip install phonemizer -
OR install it from sources with:
git clone https://github.com/bootphon/phonemizer cd phonemizer python setup.py installIf you experiment an error such as
ImportError: No module named setuptoolsduring installation, refeer to issue #11.
Alternatively you can run the phonemizer within docker, using the
provided Dockerfile. To build the docker image, have a:
git clone https://github.com/bootphon/phonemizer
cd phonemizer
sudo docker build -t phonemizer .Then run an interactive session with:
sudo docker run -it phonemizer /bin/bashWhen installed from sources or whithin a Docker image, you can run the tests
suite from the root phonemizer folder (once you installed pytest):
pip install pytest
pytestThe phonemizer project is open-source and is welcoming contributions from
everyone. Please look at the contributors guidelines if you
wish to contribute.
To refenrece the phonemizer in your own work, please cite the following JOSS
paper.
@article{Bernard2021,
doi = {10.21105/joss.03958},
url = {https://doi.org/10.21105/joss.03958},
year = {2021},
publisher = {The Open Journal},
volume = {6},
number = {68},
pages = {3958},
author = {Mathieu Bernard and Hadrien Titeux},
title = {Phonemizer: Text to Phones Transcription for Multiple Languages in Python},
journal = {Journal of Open Source Software}
}In Python import the phonemize function with from phonemizer import phonemize. See the function documentation.
It is much more efficient to minimize the number of calls to the phonemize
function. Indeed the initialization of the phonemization backend can be
expensive, especially for espeak. In one example:
from phonemizer import phonemize
text = [line1, line2, ...]
# Do this:
phonemized = phonemize(text, ...)
# Not this:
phonemized = [phonemize(line, ...) for line in text]
# An alternative is to directly instanciate the backend and to call the
# phonemize function from it:
from phonemizer.backend import EspeakBackend
backend = EspeakBackend('en-us', ...)
phonemized = [backend.phonemize(line, ...) for line in text]The following example downloads a text and phonemizes it using the festival
backend, preserving punctuation and using 4 jobs in parallel. The phones are not
separated, words are separated by a space and syllables by |.
# need to pip install requests
import requests
from phonemizer import phonemize
from phonemizer.separator import Separator
# text is a list of 190 English sentences downloaded from github
url = (
'https://gist.githubusercontent.com/CorentinJ/'
'0bc27814d93510ae8b6fe4516dc6981d/raw/'
'bb6e852b05f5bc918a9a3cb439afe7e2de570312/small_corpus.txt')
text = requests.get(url).content.decode()
text = [line.strip() for line in text.split('\n') if line]
# phn is a list of 190 phonemized sentences
phn = phonemize(
text,
language='en-us',
backend='festival',
separator=Separator(phone=None, word=' ', syllable='|'),
strip=True,
preserve_punctuation=True,
njobs=4)The following example extracts a list of words present in a text, ignoring
punctuation, and builds a dictionary word: [phones], e.g. {'students': 's t uː d ə n t s', 'cobb': 'k ɑː b', 'its': 'ɪ t s', 'put': 'p ʊ t', ...}. We
consider here the same text as in the previous example.
from phonemizer.backend import EspeakBackend
from phonemizer.punctuation import Punctuation
from phonemizer.separator import Separator
# remove all the punctuation from the text, considering only the specified
# punctuation marks
text = Punctuation(';:,.!"?()-').remove(text)
# build the set of all the words in the text
words = {w.lower() for line in text for w in line.strip().split(' ') if w}
# initialize the espeak backend for English
backend = EspeakBackend('en-us')
# separate phones by a space and ignoring words boundaries
separator = Separator(phone=' ', word=None)
# build the lexicon by phonemizing each word one by one. The backend.phonemize
# function expect a list as input and outputs a list.
lexicon = {
word: backend.phonemize([word], separator=separator, strip=True)[0]
for word in words}The above examples can be run from Python using the phonemize function
For a complete list of available options, have a:
phonemize --helpSee the installed backends with the --version option:
$ phonemize --version
phonemizer-3.0
available backends: espeak-ng-1.50, espeak-mbrola, festival-2.5.0, segments-2.1.3-
from stdin to stdout:
$ echo "hello world" | phonemize həloʊ wɜːld
-
Prepend the input text to output:
$ echo "hello world" | phonemize --prepend-text hello world | həloʊ wɜːld $ echo "hello world" | phonemize --prepend-text=';' hello world ; həloʊ wɜːld
-
from file to stdout
$ echo "hello world" > hello.txt $ phonemize hello.txt həloʊ wɜːld
-
from file to file
$ phonemize hello.txt -o hello.phon --strip $ cat hello.phon həloʊ wɜːld
-
The default is to use espeak us-english:
$ echo "hello world" | phonemize həloʊ wɜːld $ echo "hello world" | phonemize -l en-us -b espeak həloʊ wɜːld $ echo 'hello world' | phonemize -l en-us -b espeak --tie həlo͡ʊ wɜːld
-
Use festival US English instead
$ echo "hello world" | phonemize -l en-us -b festival hhaxlow werld
-
In French, using espeak and espeak-mbrola, with custom token separators (see below). espeak-mbrola does not support words separation.
$ echo "bonjour le monde" | phonemize -b espeak -l fr-fr -p ' ' -w '/w ' b ɔ̃ ʒ u ʁ /w l ə /w m ɔ̃ d /w $ echo "bonjour le monde" | phonemize -b espeak-mbrola -l mb-fr1 -p ' ' -w '/w ' b o~ Z u R l @ m o~ d
-
In Japanese, using segments
$ echo 'konnichiwa' | phonemize -b segments -l japanese konnitʃiwa $ echo 'konnichiwa' | phonemize -b segments -l ./phonemizer/share/japanese.g2p konnitʃiwa
The exhaustive list of supported languages is available with the command
phonemize --list-languages [--backend <backend>].
-
Languages supported by espeak are available here.
-
Languages supported by espeak-mbrola are available here. Please note that the mbrola voices are not bundled with the phonemizer nor the mbrola binary and must be installed separately.
-
Languages supported by festival are:
en-us -> english-us -
Languages supported by the segments backend are:
chintang -> ./phonemizer/share/segments/chintang.g2p cree -> ./phonemizer/share/segments/cree.g2p inuktitut -> ./phonemizer/share/segments/inuktitut.g2p japanese -> ./phonemizer/share/segments/japanese.g2p sesotho -> ./phonemizer/share/segments/sesotho.g2p yucatec -> ./phonemizer/share/segments/yucatec.g2pInstead of a language you can also provide a file specifying a grapheme to phone mapping (see the files above for examples).
You can specify separators for phones, syllables (festival only) and words (excepted espeak-mbrola).
$ echo "hello world" | phonemize -b festival -w ' ' -p ''
hhaxlow werld
$ echo "hello world" | phonemize -b festival -p ' ' -w ''
hh ax l ow w er l d
$ echo "hello world" | phonemize -b festival -p '-' -s '|'
hh-ax-l-|ow-| w-er-l-d-|
$ echo "hello world" | phonemize -b festival -p '-' -s '|' --strip
hh-ax-l|ow w-er-l-d
$ echo "hello world" | phonemize -b festival -p ' ' -s ';esyll ' -w ';eword '
hh ax l ;esyll ow ;esyll ;eword w er l d ;esyll ;ewordYou cannot specify the same separator for several tokens (for instance a space for both phones and words):
$ echo "hello world" | phonemize -b festival -p ' ' -w ' '
fatal error: illegal separator with word=" ", syllable="" and phone=" ",
must be all differents if not emptyBy default the punctuation is removed in the phonemized output. You can preserve
it using the --preserve-punctuation option (not supported by the
espeak-mbrola backend):
$ echo "hello, world!" | phonemize --strip
həloʊ wɜːld
$ echo "hello, world!" | phonemize --preserve-punctuation --strip
həloʊ, wɜːld!The default punctuation marks are each of the following characters: ;:,.!?¡¿—…"«»“”.
These can be overridden by the --punctuation-marks option.
$ echo "hello, world!" | phonemize --preserve-punctuation --strip --punctuation-marks '!?'
həloʊ wɜːld!The punctuation marks can be specified as a regular expression by additionally using the
--punctuation-marks-is-regex option. For example, to preserve the default punctuation marks
except for commas and periods in the middle of numbers, the following will work:
$ echo "1,000, or so." | phonemize --preserve-punctuation --strip --punctuation-marks '[;:!?¡¿—…"«»“”]|[,.](?!\d)' --punctuation-marks-is-regex
wʌn θaʊzənd, ɔːɹ soʊ.-
The espeak backend can output the stresses on phones:
$ echo "hello world" | phonemize -l en-us -b espeak --with-stress həlˈoʊ wˈɜːld
-
The espeak backend can add tie on multi-characters phonemes:
$ echo "hello world" | phonemize -l en-us -b espeak --tie həlo͡ʊ wɜːld
-
⚠️ The espeak backend can switch languages during phonemization (below from French to English), use the--language-switchoption to deal with it:$ echo "j'aime le football" | phonemize -l fr-fr -b espeak --language-switch keep-flags [WARNING] fount 1 utterances containing language switches on lines 1 [WARNING] extra phones may appear in the "fr-fr" phoneset [WARNING] language switch flags have been kept (applying "keep-flags" policy) ʒɛm lə- (en)fʊtbɔːl(fr) $ echo "j'aime le football" | phonemize -l fr-fr -b espeak --language-switch remove-flags [WARNING] fount 1 utterances containing language switches on lines 1 [WARNING] extra phones may appear in the "fr-fr" phoneset [WARNING] language switch flags have been removed (applying "remove-flags" policy) ʒɛm lə- fʊtbɔːl $ echo "j'aime le football" | phonemize -l fr-fr -b espeak --language-switch remove-utterance [WARNING] removed 1 utterances containing language switches (applying "remove-utterance" policy)
-
⚠️ The espeak backend sometimes merge words together in the output, use the--words-mismatchoption to deal with it:$ echo "that's it, words are merged" | phonemize -l en-us -b espeak [WARNING] words count mismatch on 100.0% of the lines (1/1) ðætsɪt wɜːdz ɑːɹ mɜːdʒd
Copyright 2015-2021 Mathieu Bernard
This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/.