demo_generator.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Accented Text-to-Speech Synthesis with a Conditional Variational Autoencoder\n",
    "\n",
    "###ICASSP 2023 Submission"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Naturalness test samples - no accent conversion vs accent conversion\n",
    "These samples were synthesized using the averaged representation of speakers and accents. First set is without accent conversion, the second (Conv) is with accent conversion.\n",
    "\n",
    "Utterance 1: He will knock you off a few sticks in no time.\n",
    "Utterance 2: I graduated last of my class.\n",
    "Utterance 3: For the twentieth time that evening the two men shook hands.\n",
    "Utterance 4: I will go over tomorrow afternoon.\n",
    "\n",
    "<table border=\"0\">\n",
    " <tr>\n",
    "    <td style=\"text-align: left\"><b style=\"font-size:14px\">Ground Truth</b></td> \n",
    "    <td style=\"text-align: left\"><b style=\"font-size:14px\">CVAE-NL</b></td>\n",
    "    <td style=\"text-align: left\"><b style=\"font-size:14px\">CVAE-L</b></td>\n",
    "    <td style=\"text-align: left\"><b style=\"font-size:14px\">GST</b></td>\n",
    "    <td style=\"text-align: left\"><b style=\"font-size:14px\">GMVAE</b></td>\n",
    "    <td style=\"text-align: left\"><b style=\"font-size:14px\">Conv CVAE-NL</b></td>\n",
    "    <td style=\"text-align: left\"><b style=\"font-size:14px\">Conv CVAE-L</b></td>\n",
    "    <td style=\"text-align: left\"><b style=\"font-size:14px\">Conv GST</b></td>\n",
    "    <td style=\"text-align: left\"><b style=\"font-size:14px\">Conv GMVAE</b></td>\n",
    " </tr>\n",
    " <tr>\n",
    "    <td>Speaker: ABA (Arabic) <audio src=\"audio/MOS/GT/ABAarctic_a0334.wav\" controls>alternative text</audio><br/>\n",
    "Speaker: HKK (Korean) <audio src=\"audio/MOS/GT/HKKarctic_a0304.wav\" controls>alternative text</audio><br/>\n",
    "Speaker: NCC (Chinese) <audio src=\"audio/MOS/GT/NCCarctic_a0003.wav\" controls>alternative text</audio><br/>\n",
    "Speaker: SVBI (Hindi) <audio src=\"audio/MOS/GT/SVBIarctic_a0390.wav\" controls>alternative text</audio>\n",
    "    </td>\n",
    "    <td><audio src=\"audio/MOS/CVAE-NL/ABAarctic_a0334.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/MOS/CVAE-NL/HKKarctic_a0304.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/MOS/CVAE-NL/NCCarctic_a0003.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/MOS/CVAE-NL/SVBIarctic_a0390.wav\" controls>alternative text</audio>\n",
    "    </td>\n",
    "    <td><audio src=\"audio/MOS/CVAE-L/ABAarctic_a0334.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/MOS/CVAE-L/HKKarctic_a0304.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/MOS/CVAE-L/NCCarctic_a0003.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/MOS/CVAE-L/SVBIarctic_a0390.wav\" controls>alternative text</audio>\n",
    "    </td>\n",
    "    <td><audio src=\"audio/MOS/GST/ABAarctic_a0334.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/MOS/GST/HKKarctic_a0304.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/MOS/GST/NCCarctic_a0003.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/MOS/GST/SVBIarctic_a0390.wav\" controls>alternative text</audio>\n",
    "    </td>\n",
    "    <td><audio src=\"audio/MOS/GMVAE/ABAarctic_a0334.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/MOS/GMVAE/HKKarctic_a0304.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/MOS/GMVAE/NCCarctic_a0003.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/MOS/GMVAE/SVBIarctic_a0390.wav\" controls>alternative text</audio>\n",
    "    </td>\n",
    "    <td><audio src=\"audio/MOS/convCVAE-NL/ABA_Korean_He will knock you off a few sticks in no time.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/MOS/convCVAE-NL/HKK_Arabic_I graduated last of my class.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/MOS/convCVAE-NL/NCC_Hindi_For the twentieth time that evening the two men shook hands.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/MOS/convCVAE-NL/SVBI_Chinese_I will go over tomorrow afternoon.wav\" controls>alternative text</audio>\n",
    "    </td>\n",
    "    <td><audio src=\"audio/MOS/convCVAE-L/ABA_Korean_He will knock you off a few sticks in no time.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/MOS/convCVAE-L/HKK_Arabic_I graduated last of my class.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/MOS/convCVAE-L/NCC_Hindi_For the twentieth time that evening the two men shook hands.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/MOS/convCVAE-L/SVBI_Chinese_I will go over tomorrow afternoon.wav\" controls>alternative text</audio>\n",
    "    </td>\n",
    "    <td><audio src=\"audio/MOS/convGST/ABA_Korean_He will knock you off a few sticks in no time.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/MOS/convGST/HKK_Arabic_I graduated last of my class.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/MOS/convGST/NCC_Hindi_For the twentieth time that evening the two men shook hands.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/MOS/convGST/SVBI_Chinese_I will go over tomorrow afternoon.wav\" controls>alternative text</audio>\n",
    "    </td>\n",
    "    <td><audio src=\"audio/MOS/convGMVAE/ABA_Korean_He will knock you off a few sticks in no time.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/MOS/convGMVAE/HKK_Arabic_I graduated last of my class.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/MOS/convGMVAE/NCC_Hindi_For the twentieth time that evening the two men shook hands.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/MOS/convGMVAE/SVBI_Chinese_I will go over tomorrow afternoon.wav\" controls>alternative text</audio>\n",
    "    </td>\n",
    " </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Accent conversion task\n",
    "These samples were converted to the target accent.\n",
    "\n",
    "Utterance 1: For the twentieth time that evening the two men shook hands.\n",
    "Utterance 2: And you always want to see it in the superlative degree.\n",
    "Utterance 3: I will go over tomorrow afternoon.\n",
    "\n",
    "<table border=\"0\">\n",
    " <tr>\n",
    "    <td style=\"text-align: left\"><b style=\"font-size:14px\">Source Ground Truth</b></td> \n",
    "    <td style=\"text-align: left\"><b style=\"font-size:14px\">CVAE-NL</b></td>\n",
    "    <td style=\"text-align: left\"><b style=\"font-size:14px\">CVAE-L</b></td>\n",
    "    <td style=\"text-align: left\"><b style=\"font-size:14px\">GST</b></td>\n",
    "    <td style=\"text-align: left\"><b style=\"font-size:14px\">GMVAE</b></td>\n",
    " </tr>\n",
    " <tr>\n",
    "     <td>Speaker:THV (Vietnamese) Accent: Arabic    <audio src=\"audio/converted/GTsource/THVarctic_a0003.wav\" controls>alternative text</audio><br/>\n",
    "         Speaker:THV (Vietnamese) Accent: Hindi   <audio src=\"audio/converted/GTsource/THVarctic_a0003.wav\" controls>alternative text</audio><br/>\n",
    "         Speaker:NCC (Chinese) Accent: Hindi       <audio src=\"audio/converted/GTsource/NCCarctic_a0007.wav\" controls>alternative text</audio><br/>\n",
    "         Speaker:NCC (Chinese) Accent: Spanish    <audio src=\"audio/converted/GTsource/NCCarctic_a0007.wav\" controls>alternative text</audio><br/>\n",
    "         Speaker:EBVS (Spanish) Accent: Chinese       <audio src=\"audio/converted/GTsource/EBVSarctic_a0007.wav\" controls>alternative text</audio><br/>\n",
    "         Speaker:EBVS (Spanish) Accent: Korean    <audio src=\"audio/converted/GTsource/EBVSarctic_a0007.wav\" controls>alternative text</audio><br/>\n",
    "         Speaker:HKK (Korean) Accent: Arabic       <audio src=\"audio/converted/GTsource/EBVSarctic_a0007.wav\" controls>alternative text</audio><br/>\n",
    "         Speaker:HKK (Korean) Accent: Spanish    <audio src=\"audio/converted/GTsource/EBVSarctic_a0007.wav\" controls>alternative text</audio>\n",
    "    </td>\n",
    "    <td><audio src=\"audio/converted/CVAE-NL/THV_Arabic_For the twentieth time that evening the two men shook hands.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/converted/CVAE-NL/THV_Hindi_For the twentieth time that evening the two men shook hands.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/converted/CVAE-NL/NCC_Hindi_And you always want to see it in the superlative degree.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/converted/CVAE-NL/NCC_Spanish_And you always want to see it in the superlative degree.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/converted/CVAE-NL/EBVS_Chinese_And you always want to see it in the superlative degree.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/converted/CVAE-NL/EBVS_Korean_And you always want to see it in the superlative degree.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/converted/CVAE-NL/HKK_Arabic_I will go over tomorrow afternoon.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/converted/CVAE-NL/HKK_Spanish_I will go over tomorrow afternoon.wav\" controls>alternative text</audio>\n",
    "    </td>     \n",
    "     <td><audio src=\"audio/converted/CVAE-L/THV_Arabic_For the twentieth time that evening the two men shook hands.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/converted/CVAE-L/THV_Hindi_For the twentieth time that evening the two men shook hands.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/converted/CVAE-L/NCC_Hindi_And you always want to see it in the superlative degree.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/converted/CVAE-L/NCC_Spanish_And you always want to see it in the superlative degree.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/converted/CVAE-L/EBVS_Chinese_And you always want to see it in the superlative degree.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/converted/CVAE-L/EBVS_Korean_And you always want to see it in the superlative degree.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/converted/CVAE-L/HKK_Arabic_I will go over tomorrow afternoon.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/converted/CVAE-L/HKK_Spanish_I will go over tomorrow afternoon.wav\" controls>alternative text</audio>\n",
    "    </td>     \n",
    "     <td><audio src=\"audio/converted/GST/THV_Arabic_For the twentieth time that evening the two men shook hands.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/converted/GST/THV_Hindi_For the twentieth time that evening the two men shook hands.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/converted/GST/NCC_Hindi_And you always want to see it in the superlative degree.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/converted/GST/NCC_Spanish_And you always want to see it in the superlative degree.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/converted/GST/EBVS_Chinese_And you always want to see it in the superlative degree.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/converted/GST/EBVS_Korean_And you always want to see it in the superlative degree.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/converted/GST/HKK_Arabic_I will go over tomorrow afternoon.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/converted/GST/HKK_Spanish_I will go over tomorrow afternoon.wav\" controls>alternative text</audio>\n",
    "    </td>     \n",
    "     <td><audio src=\"audio/converted/GMVAE/THV_Arabic_For the twentieth time that evening the two men shook hands.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/converted/GMVAE/THV_Hindi_For the twentieth time that evening the two men shook hands.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/converted/GMVAE/NCC_Hindi_And you always want to see it in the superlative degree.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/converted/GMVAE/NCC_Spanish_And you always want to see it in the superlative degree.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/converted/GMVAE/EBVS_Chinese_And you always want to see it in the superlative degree.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/converted/GMVAE/EBVS_Korean_And you always want to see it in the superlative degree.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/converted/GMVAE/HKK_Arabic_I will go over tomorrow afternoon.wav\" controls>alternative text</audio><br/>\n",
    "        <audio src=\"audio/converted/GMVAE/HKK_Spanish_I will go over tomorrow afternoon.wav\" controls>alternative text</audio>\n",
    "    </td>     \n",
    " </tr>\n",
    "</table>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}