main.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ae40f825-c843-4462-86ce-7a8d273a1da5",
   "metadata": {},
   "source": [
    "# **AI Storytelling**\n",
    "\n",
    "<img src=\"assets/logo.jpg\" width=\"500\" height=\"300\">\n",
    "\n",
    "This is Natural Language Processing platform that targets on conversion of short stories to audiobooks with features:\n",
    "\n",
    "1.\tSeparated text into dialogues and narrations.\n",
    "2.\tReplaced pronouns by nouns in narrations.\n",
    "3.\tExtracted characters from narration.\n",
    "4.\tIdentified genders of characters using Gaussian NB with accuracy of 87%.\n",
    "5.\tPredicted age of characters using Random Forest Regressor with RMSE of 14.6.\n",
    "6.\tFound emotions of characters from dialogues using Transformers Pipeline.\n",
    "7.\tFound pitch, tempo and loudness based on emotions, age and gender.\n",
    "8.\tPerformed audio processing for converting dialogues and narrations into audios.\n",
    "9.\tCombined audios to create the complete audiobook. \n",
    "10.\tFinal software development.\r\n",
    "\n",
    "\n",
    "Based on the success of the project, it could be extended with image or video processing features in upcoming days."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11afbf11-2727-4d5e-9797-2af2363c8ebd",
   "metadata": {},
   "source": [
    "### One-time Installations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6bf23504-fa70-4a70-b25b-efb292fadefd",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -r requirements.txt\n",
    "!python -m spacy download en_core_web_lg"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9ebde27-b693-4ae4-8fbb-2779859d3ee9",
   "metadata": {},
   "source": [
    "### Import Libraries and Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "id": "274ae554-7c39-45ea-ad21-3a8fe63eb609",
   "metadata": {},
   "outputs": [],
   "source": [
    "# basic libraries\n",
    "import re\n",
    "import os\n",
    "import random\n",
    "import string\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "ff82950a-e67c-443a-a8b3-b554678b0248",
   "metadata": {},
   "outputs": [],
   "source": [
    "# nlp libraries\n",
    "import spacy\n",
    "nlp = spacy.load(\"en_core_web_lg\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "851f8a25-1e2f-4c63-b249-ffecf61b64b0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ignore warnings\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "dcfe2c09-b858-4141-86e8-bc6e23df44f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# nlp libraries\n",
    "import nltk\n",
    "# nltk.download('wordnet')\n",
    "# nltk.download(\"stopwords\")\n",
    "from nltk.corpus import stopwords, wordnet\n",
    "from nltk.stem import WordNetLemmatizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "9d8cbfd1-cbdd-40e4-95e5-badf257b7326",
   "metadata": {},
   "outputs": [],
   "source": [
    "# emotion detection libraries\n",
    "from transformers import pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "id": "6cfb5911-4127-46bc-b726-ca41ab5c83b2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# text-to-speech libraries\n",
    "import pyttsx3\n",
    "from gtts import gTTS\n",
    "from moviepy.editor import concatenate_audioclips, AudioFileClip\n",
    "import librosa\n",
    "import soundfile as sf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "d177de12-eec7-41be-af71-cc9c7708c1cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# other libraries\n",
    "import joblib"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24ecda03-398b-4b9e-96f0-af5014502eca",
   "metadata": {},
   "source": [
    "### Files and Dataframes Defining"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "0a7451d1-9491-47b2-9c8a-0da6cb90b3bc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['beautiful garden.txt',\n",
       " 'beneath the tides.txt',\n",
       " 'cafe at midnight.txt',\n",
       " 'the painted door.txt',\n",
       " 'threads of imagination.txt']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# checking stories\n",
    "os.listdir(\"stories\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "334267e2-b696-4a16-b1db-1499788eda42",
   "metadata": {},
   "outputs": [
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      "Enter text file name: beautiful garden\n"
     ]
    }
   ],
   "source": [
    "# input title\n",
    "title = input(\"Enter text file name:\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "0cf9b1d2-2363-4781-86e4-cccbae5f948d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# opening text file\n",
    "story = f\"stories/{title}.txt\"\n",
    "with open(story, \"r\") as f:\n",
    "    text = f.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "7f3902d3-df6d-47b8-9bc4-e2d577694e9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create dataframes\n",
    "df_characters = pd.DataFrame(columns=[\"cid\", \"name\", \"num_dialogue\", \"num_occurrence\"])\n",
    "df_lines = pd.DataFrame(columns=[\"pid\", \"ndid\", \"character\", \"dialogue\", \"narration\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc5ec1fc-e4af-4ad6-ae2c-e159f746cff2",
   "metadata": {},
   "source": [
    "# 1. Lines Identification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "ab1ac800-2b5a-422c-948b-4288593aa33b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# converting text to paragraphs\n",
    "paragraphs = text.split(\"\\n\")\n",
    "non_empty_paragraphs = list(filter(lambda x: x != '', paragraphs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "bcf6ec4d-f0e3-40a8-9d54-5a7af7e4f340",
   "metadata": {},
   "outputs": [],
   "source": [
    "# function to identify narrations and dialogues\n",
    "def identify_narrations_and_dialogues(paragraph):\n",
    "    \"\"\"\n",
    "    :param paragraph: string of paragraph in a story\n",
    "    :return: list of tuples in (id, name_of_speaker, dialogue, narration) format\n",
    "    \"\"\"\n",
    "    divisions = paragraph.split('\"')\n",
    "    divisions = list(filter(lambda x: x != '', divisions))\n",
    "    i, spoken = 0, False\n",
    "    narrations_and_dialogues = []\n",
    "    for division in divisions:\n",
    "        start_index = paragraph.find(division)\n",
    "        end_index = start_index + len(division) - 1\n",
    "        try:\n",
    "            if '\"'==paragraph[start_index-1:start_index] and '\"'==paragraph[end_index+1:end_index+2] and spoken==False:\n",
    "                # dialogues\n",
    "                narrations_and_dialogues.append((i, None, division, None))\n",
    "                spoken = True\n",
    "            else:\n",
    "                # narrations\n",
    "                narrations_and_dialogues.append((i, None, None, division))\n",
    "                spoken = False\n",
    "        except:\n",
    "            # narrations\n",
    "            narrations_and_dialogues.append((i, None, None, division))\n",
    "            spoken = False\n",
    "        i += 1\n",
    "    return narrations_and_dialogues\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "3434c2cc-cd18-419e-90af-e95a6d2ef819",
   "metadata": {},
   "outputs": [],
   "source": [
    "# identifying lines (narrations or dialogues) from each paragraphs\n",
    "pid_num = 0\n",
    "for paragraph in non_empty_paragraphs:\n",
    "    for row in [(pid_num,)+nad for nad in identify_narrations_and_dialogues(paragraph)]:\n",
    "        df_lines = df_lines._append(pd.Series(row, index=df_lines.columns), ignore_index=True)\n",
    "    pid_num += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "ba4a3201-ffb6-4eda-92ff-371db814c596",
   "metadata": {},
   "outputs": [],
   "source": [
    "# set index using paragraph and nd\n",
    "df_lines.set_index(['pid', 'ndid'], inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "6f31eb55-9b97-4760-ab64-8c9a7fa25b37",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>character</th>\n",
       "      <th>dialogue</th>\n",
       "      <th>narration</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pid</th>\n",
       "      <th>ndid</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Once upon a time, in a small, quiet village, t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>One sunny afternoon, as Lily was by the river,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">2</th>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Unbeknownst to Lily, a kind stranger had been ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>None</td>\n",
       "      <td>You have a heart as beautiful as that butterfl...</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">3</th>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Lily blushed, not used to receiving compliment...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>None</td>\n",
       "      <td>I've been searching for someone just like you....</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">4</th>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Lily's eyes sparkled with excitement. She had ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>None</td>\n",
       "      <td>I'd love to help,</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>she replied.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>From that day on, Lily spent her afternoons ca...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         character                                           dialogue  \\\n",
       "pid ndid                                                                \n",
       "0   0         None                                               None   \n",
       "1   0         None                                               None   \n",
       "2   0         None                                               None   \n",
       "    1         None  You have a heart as beautiful as that butterfl...   \n",
       "3   0         None                                               None   \n",
       "    1         None  I've been searching for someone just like you....   \n",
       "4   0         None                                               None   \n",
       "    1         None                                  I'd love to help,   \n",
       "    2         None                                               None   \n",
       "5   0         None                                               None   \n",
       "\n",
       "                                                  narration  \n",
       "pid ndid                                                     \n",
       "0   0     Once upon a time, in a small, quiet village, t...  \n",
       "1   0     One sunny afternoon, as Lily was by the river,...  \n",
       "2   0     Unbeknownst to Lily, a kind stranger had been ...  \n",
       "    1                                                  None  \n",
       "3   0     Lily blushed, not used to receiving compliment...  \n",
       "    1                                                  None  \n",
       "4   0     Lily's eyes sparkled with excitement. She had ...  \n",
       "    1                                                  None  \n",
       "    2                                          she replied.  \n",
       "5   0     From that day on, Lily spent her afternoons ca...  "
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# narrations and dialogues\n",
    "df_lines.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2bde477c-1d1c-4968-b289-48fba5c8cf0f",
   "metadata": {},
   "source": [
    "# 2. Characters Identification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "d7bfcacb-ffe7-446a-a152-c4e1a3a0bf0a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# function to find subjects speaking corresponding dialogues\n",
    "def identify_subjects(sentence):\n",
    "    \"\"\"\n",
    "    :param sentence: string of sentence in the story\n",
    "    :return: individual speaker\n",
    "    \"\"\"\n",
    "    tokens = []\n",
    "    individual, individuals = '', []\n",
    "    \n",
    "    for tok in reversed(nlp(sentence)):\n",
    "        if str(tok)=='.' and tok.dep_ == 'punct':\n",
    "            break\n",
    "        tokens.append([tok, tok.dep_])\n",
    "    tokens = tokens[::-1]\n",
    "    \n",
    "    for i in range(len(tokens)-1, -1, -1):\n",
    "        if tokens[i][1] == \"nsubj\":\n",
    "            individual = str(tokens[i][0])\n",
    "        elif (tokens[i][1] in ['compound', 'det'] and individual != ''):\n",
    "            individual = str(tokens[i][0]) + \" \" + individual\n",
    "        else:\n",
    "            individuals.append(individual)\n",
    "            \n",
    "    individuals.append(individual)\n",
    "    return list(set([ind for ind in individuals if ind != '']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "f7dd6bd6-95a5-490c-81c3-01eef1cec37a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# finding semantic similarity\n",
    "def calculating_semantic_similarity(subject, sentence):\n",
    "    \"\"\"\n",
    "    :param subject: list of strings available in a sentence\n",
    "    :return: similarity score\n",
    "    \"\"\"\n",
    "    subject_doc = nlp(subject)\n",
    "    sentence_doc = nlp(sentence)\n",
    "    \n",
    "    subject_vec = subject_doc.vector\n",
    "    sentence_vec = sentence_doc.vector\n",
    "    \n",
    "    subject_vec_norm = subject_doc.vector_norm\n",
    "    sentence_vec_norm = sentence_doc.vector_norm\n",
    "    \n",
    "    if subject_vec_norm > 0 and sentence_vec_norm > 0:\n",
    "        similarity_score = subject_vec.dot(sentence_vec) / (subject_vec_norm * sentence_vec_norm)\n",
    "    else:\n",
    "        similarity_score = 0\n",
    "    \n",
    "    return similarity_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "83b3f999-a9fa-482f-ad4e-509f8c981d55",
   "metadata": {},
   "outputs": [],
   "source": [
    "# finding subjects from dialogues and narrations\n",
    "dlg_list = df_lines.index[df_lines['dialogue'].notna()].tolist()\n",
    "\n",
    "for dlgi in dlg_list:\n",
    "    if df_lines.loc[(dlgi[0], dlgi[1]), 'dialogue'][-1] == ',':\n",
    "        full_narration = df_lines.loc[(dlgi[0], dlgi[1]+1), 'narration']\n",
    "        final_narration = full_narration.rstrip().rstrip(string.punctuation)\n",
    "        subjects = identify_subjects(final_narration)\n",
    "        relevant_dictionary = dict((subject.lower(), calculating_semantic_similarity(subject.lower(), df_lines.loc[(dlgi[0], dlgi[1]+1), 'narration'].lower())) for subject in subjects)\n",
    "        relevant_subject = max(relevant_dictionary, key=lambda k: relevant_dictionary[k]) if relevant_dictionary else None\n",
    "    else:\n",
    "        full_narration = df_lines.loc[(dlgi[0], dlgi[1]-1), 'narration']\n",
    "        final_narration = full_narration.rstrip().rstrip(string.punctuation)\n",
    "        subjects = identify_subjects(final_narration)\n",
    "        relevant_dictionary = dict((subject.lower(), calculating_semantic_similarity(subject.lower(), df_lines.loc[(dlgi[0], dlgi[1]-1), 'narration'].lower())) for subject in subjects)\n",
    "        relevant_subject = max(relevant_dictionary, key=lambda k: relevant_dictionary[k]) if relevant_dictionary else None\n",
    "    df_lines.loc[(dlgi[0], dlgi[1]), 'character'] = relevant_subject"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "a3d8cd9a-2faa-4648-9da4-7fe0a1c96fc2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>character</th>\n",
       "      <th>dialogue</th>\n",
       "      <th>narration</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pid</th>\n",
       "      <th>ndid</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Once upon a time, in a small, quiet village, t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>One sunny afternoon, as Lily was by the river,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">2</th>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Unbeknownst to Lily, a kind stranger had been ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>the stranger</td>\n",
       "      <td>You have a heart as beautiful as that butterfl...</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">3</th>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Lily blushed, not used to receiving compliment...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mrs. thompson</td>\n",
       "      <td>I've been searching for someone just like you....</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">4</th>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Lily's eyes sparkled with excitement. She had ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>she</td>\n",
       "      <td>I'd love to help,</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>she replied.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>From that day on, Lily spent her afternoons ca...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              character                                           dialogue  \\\n",
       "pid ndid                                                                     \n",
       "0   0              None                                               None   \n",
       "1   0              None                                               None   \n",
       "2   0              None                                               None   \n",
       "    1      the stranger  You have a heart as beautiful as that butterfl...   \n",
       "3   0              None                                               None   \n",
       "    1     mrs. thompson  I've been searching for someone just like you....   \n",
       "4   0              None                                               None   \n",
       "    1               she                                  I'd love to help,   \n",
       "    2              None                                               None   \n",
       "5   0              None                                               None   \n",
       "\n",
       "                                                  narration  \n",
       "pid ndid                                                     \n",
       "0   0     Once upon a time, in a small, quiet village, t...  \n",
       "1   0     One sunny afternoon, as Lily was by the river,...  \n",
       "2   0     Unbeknownst to Lily, a kind stranger had been ...  \n",
       "    1                                                  None  \n",
       "3   0     Lily blushed, not used to receiving compliment...  \n",
       "    1                                                  None  \n",
       "4   0     Lily's eyes sparkled with excitement. She had ...  \n",
       "    1                                                  None  \n",
       "    2                                          she replied.  \n",
       "5   0     From that day on, Lily spent her afternoons ca...  "
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# dialogues and corresponding characters\n",
    "df_lines.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "da002af1-055a-46a6-a7c9-4321800f0588",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([None, 'the stranger', 'mrs. thompson', 'she', 'that'], dtype=object)"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check characters\n",
    "df_lines.character.unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "44c4edd3-d8a3-4956-9567-e66a953948a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# gathering characters from df_lines\n",
    "characters = df_lines['character'].value_counts()\n",
    "df_characters['name'] = characters.index\n",
    "df_characters['num_dialogue'] = characters.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "4816a1d8-b78f-42cb-87d6-d99b462bd3dd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# declaring cid as index\n",
    "df_characters['cid'] = range(0, len(df_characters))\n",
    "df_characters.set_index('cid', inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "1ca70cc9-ce19-4834-afbb-9db21c37cf88",
   "metadata": {},
   "outputs": [],
   "source": [
    "# counting number of times these names occurred in text\n",
    "df_characters['num_occurrence'] = df_characters['name'].apply(lambda x: len(re.findall(x, text.lower())))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "c8207497-45ec-4201-8606-75cd600b0c5a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>num_dialogue</th>\n",
       "      <th>num_occurrence</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cid</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>the stranger</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mrs. thompson</td>\n",
       "      <td>1</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>she</td>\n",
       "      <td>1</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>that</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              name  num_dialogue  num_occurrence\n",
       "cid                                             \n",
       "0     the stranger             1               1\n",
       "1    mrs. thompson             1               8\n",
       "2              she             1              11\n",
       "3             that             1               6"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# top 10 characters\n",
    "df_characters.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a473698c-bf0a-4161-be02-832b5c84a6e3",
   "metadata": {},
   "source": [
    "# 3. Cleaned Dialogue Generation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "f0f4ef4a-e75d-4584-aaf5-a9ef94ba276b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# defining stopwords set and lemma\n",
    "set_of_stopwords = set(stopwords.words(\"english\"))\n",
    "lemma = WordNetLemmatizer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "6a6992ba-0081-4daf-ae6e-5f999477b0f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# function to remove contractions and replace by spaces\n",
    "def remove_contractions(text):\n",
    "\n",
    "    contractions = [\"ain't\", \"aren't\", \"can't\", \"can't've\", \"'cause\", \"could've\", \"couldn't\", \"couldn't've\", \"didn't\", \"doesn't\", \"don't\",\n",
    "                    \"hadn't\", \"hadn't've\", \"hasn't\", \"haven't\", \"he'd\", \"he'd've\", \"he'll\", \"he'll've\", \"he's\", \"how'd\", \"how'd'y\", \"how'll\", \"how's\",\n",
    "                    \"I'd\", \"I'd've\", \"I'll\", \"I'll've\", \"I'm\", \"I've\", \"isn't\", \"it'd\", \"it'd've\", \"it'll\", \"it'll've\", \"it's\", \"let's\",\n",
    "                    \"ma'am\", \"mayn't\", \"might've\", \"mightn't\", \"mightn't've\", \"must've\", \"mustn't\", \"mustn't've\", \"needn't\", \"needn't've\",\n",
    "                    \"o'clock\", \"oughtn't\", \"oughtn't've\", \"shan't\", \"sha'n't\", \"shan't've\", \"she'd\", \"she'd've\", \"she'll\", \"she'll've\", \"she's\",\n",
    "                    \"should've\", \"shouldn't\", \"shouldn't've\", \"so've\", \"so's\", \"that'd\", \"that'd've\", \"that's\", \"there'd\", \"there'd've\", \"there's\",\n",
    "                    \"they'd\", \"they'd've\", \"they'll\", \"they'll've\", \"they're\", \"they've\", \"to've\", \"wasn't\",\n",
    "                    \"we'd\", \"we'd've\", \"we'll\", \"we'll've\", \"we're\", \"we've\", \"weren't\", \"what'll\", \"what'll've\", \"what're\", \"what's\", \"what've\",\n",
    "                    \"when's\", \"when've\", \"where'd\", \"where's\", \"where've\", \"who'll\", \"who'll've\", \"who's\", \"who've\", \"why's\", \"why've\",\n",
    "                    \"will've\", \"won't\", \"won't've\", \"would've\", \"wouldn't\", \"wouldn't've\", \"y'all\", \"y'all'd\", \"y'all'd've\", \"y'all're\", \"y'all've\",\n",
    "                    \"you'd\", \"you'd've\", \"you'll\", \"you'll've\", \"you're\", \"you've\", \"gonna\"]\n",
    "\n",
    "    for contraction in contractions:\n",
    "        text = text.replace(contraction, \" \")\n",
    "\n",
    "    return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "d0023ced-46d5-4379-acff-c1365206a3ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "# function to remove punctuations\n",
    "def remove_punctuations(text):\n",
    "    text = re.sub(r'[^\\w\\s]', '', text)\n",
    "    return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "fdb61aaa-acb9-4ee3-aa67-691183099b41",
   "metadata": {},
   "outputs": [],
   "source": [
    "# function to check noun\n",
    "def is_noun(word):\n",
    "    synsets = wordnet.synsets(word)\n",
    "    for synset in synsets:\n",
    "        if synset.pos() == 'n':\n",
    "            return True\n",
    "    return False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "9eb9bca4-89f2-4d86-a029-39a8fbbaaf2a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# dialogue to cleaned dialogue\n",
    "def cleaned_dialogue(text):\n",
    "    if text is None:\n",
    "        return ''\n",
    "    contractionless_text = remove_contractions(text)\n",
    "    punctuationless_text = remove_punctuations(contractionless_text)\n",
    "    tokens = nltk.word_tokenize(punctuationless_text)\n",
    "    filtered_words = [token.lower() for token in tokens if token.lower() not in set_of_stopwords]\n",
    "    lemmatized_words = [lemma.lemmatize(word) for word in filtered_words]\n",
    "    words = [word for word in lemmatized_words if is_noun(word)]\n",
    "    return ' '.join(words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "e1ff5755-0327-4aed-acc2-60ef7616b5ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "# list of words for dialogue\n",
    "dialogue_list = []\n",
    "for dialogue in df_lines.dialogue:\n",
    "    dialogue_list.append(cleaned_dialogue(dialogue))\n",
    "df_lines['cleaned_dialogue'] = dialogue_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "fef42fd9-dc01-44c7-b8df-52cd34e858af",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>character</th>\n",
       "      <th>dialogue</th>\n",
       "      <th>narration</th>\n",
       "      <th>cleaned_dialogue</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pid</th>\n",
       "      <th>ndid</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Once upon a time, in a small, quiet village, t...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>One sunny afternoon, as Lily was by the river,...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">2</th>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Unbeknownst to Lily, a kind stranger had been ...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>the stranger</td>\n",
       "      <td>You have a heart as beautiful as that butterfl...</td>\n",
       "      <td>None</td>\n",
       "      <td>heart butterfly dear</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">3</th>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Lily blushed, not used to receiving compliment...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mrs. thompson</td>\n",
       "      <td>I've been searching for someone just like you....</td>\n",
       "      <td>None</td>\n",
       "      <td>someone like garden full flower need help tending</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">4</th>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Lily's eyes sparkled with excitement. She had ...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>she</td>\n",
       "      <td>I'd love to help,</td>\n",
       "      <td>None</td>\n",
       "      <td>love help</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>she replied.</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>From that day on, Lily spent her afternoons ca...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              character                                           dialogue  \\\n",
       "pid ndid                                                                     \n",
       "0   0              None                                               None   \n",
       "1   0              None                                               None   \n",
       "2   0              None                                               None   \n",
       "    1      the stranger  You have a heart as beautiful as that butterfl...   \n",
       "3   0              None                                               None   \n",
       "    1     mrs. thompson  I've been searching for someone just like you....   \n",
       "4   0              None                                               None   \n",
       "    1               she                                  I'd love to help,   \n",
       "    2              None                                               None   \n",
       "5   0              None                                               None   \n",
       "\n",
       "                                                  narration  \\\n",
       "pid ndid                                                      \n",
       "0   0     Once upon a time, in a small, quiet village, t...   \n",
       "1   0     One sunny afternoon, as Lily was by the river,...   \n",
       "2   0     Unbeknownst to Lily, a kind stranger had been ...   \n",
       "    1                                                  None   \n",
       "3   0     Lily blushed, not used to receiving compliment...   \n",
       "    1                                                  None   \n",
       "4   0     Lily's eyes sparkled with excitement. She had ...   \n",
       "    1                                                  None   \n",
       "    2                                          she replied.   \n",
       "5   0     From that day on, Lily spent her afternoons ca...   \n",
       "\n",
       "                                           cleaned_dialogue  \n",
       "pid ndid                                                     \n",
       "0   0                                                        \n",
       "1   0                                                        \n",
       "2   0                                                        \n",
       "    1                                  heart butterfly dear  \n",
       "3   0                                                        \n",
       "    1     someone like garden full flower need help tending  \n",
       "4   0                                                        \n",
       "    1                                             love help  \n",
       "    2                                                        \n",
       "5   0                                                        "
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# top 10 rows\n",
    "df_lines.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6189406-eb05-4fef-9b06-bc6dc2a76fda",
   "metadata": {},
   "source": [
    "# 4. Age Detection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "a2d866cb-6819-4a57-828e-4d1ff89b82e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# loading age count vectorizer\n",
    "loaded_age_cv = joblib.load('attributes/age/age_cv.joblib')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "d5cb31a0-2730-43e7-88a3-d140e49df12d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# creating sparse matrix for transformed values and array\n",
    "age_sparse_matrix = loaded_age_cv.transform(df_lines['cleaned_dialogue'])\n",
    "age_sparse_matrix_array = age_sparse_matrix.toarray()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "00e501d3-a8b1-4d58-a7a4-b6cc472e0a79",
   "metadata": {},
   "outputs": [],
   "source": [
    "# loading age model\n",
    "loaded_age_model = joblib.load('attributes/age/age_model.joblib')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "232b0037-f1f6-4112-ad86-5bc796855255",
   "metadata": {},
   "outputs": [],
   "source": [
    "# defining X and predicting\n",
    "df_lines['age'] = loaded_age_model.predict(age_sparse_matrix_array)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "1cf2493f-1552-486d-b1c5-0ae5dec15bad",
   "metadata": {},
   "outputs": [],
   "source": [
    "# define age as None if dialogue is None\n",
    "df_lines['age'] = df_lines.apply(lambda row: None if pd.isna(row['dialogue']) else row['age'], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "d42a4e2c-7aa8-4dd8-9730-a9070b35afe8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>character</th>\n",
       "      <th>dialogue</th>\n",
       "      <th>narration</th>\n",
       "      <th>cleaned_dialogue</th>\n",
       "      <th>age</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pid</th>\n",
       "      <th>ndid</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Once upon a time, in a small, quiet village, t...</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>One sunny afternoon, as Lily was by the river,...</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">2</th>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Unbeknownst to Lily, a kind stranger had been ...</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>the stranger</td>\n",
       "      <td>You have a heart as beautiful as that butterfl...</td>\n",
       "      <td>None</td>\n",
       "      <td>heart butterfly dear</td>\n",
       "      <td>5.02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">3</th>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Lily blushed, not used to receiving compliment...</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mrs. thompson</td>\n",
       "      <td>I've been searching for someone just like you....</td>\n",
       "      <td>None</td>\n",
       "      <td>someone like garden full flower need help tending</td>\n",
       "      <td>6.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">4</th>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Lily's eyes sparkled with excitement. She had ...</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>she</td>\n",
       "      <td>I'd love to help,</td>\n",
       "      <td>None</td>\n",
       "      <td>love help</td>\n",
       "      <td>5.27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>she replied.</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>From that day on, Lily spent her afternoons ca...</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              character                                           dialogue  \\\n",
       "pid ndid                                                                     \n",
       "0   0              None                                               None   \n",
       "1   0              None                                               None   \n",
       "2   0              None                                               None   \n",
       "    1      the stranger  You have a heart as beautiful as that butterfl...   \n",
       "3   0              None                                               None   \n",
       "    1     mrs. thompson  I've been searching for someone just like you....   \n",
       "4   0              None                                               None   \n",
       "    1               she                                  I'd love to help,   \n",
       "    2              None                                               None   \n",
       "5   0              None                                               None   \n",
       "\n",
       "                                                  narration  \\\n",
       "pid ndid                                                      \n",
       "0   0     Once upon a time, in a small, quiet village, t...   \n",
       "1   0     One sunny afternoon, as Lily was by the river,...   \n",
       "2   0     Unbeknownst to Lily, a kind stranger had been ...   \n",
       "    1                                                  None   \n",
       "3   0     Lily blushed, not used to receiving compliment...   \n",
       "    1                                                  None   \n",
       "4   0     Lily's eyes sparkled with excitement. She had ...   \n",
       "    1                                                  None   \n",
       "    2                                          she replied.   \n",
       "5   0     From that day on, Lily spent her afternoons ca...   \n",
       "\n",
       "                                           cleaned_dialogue   age  \n",
       "pid ndid                                                           \n",
       "0   0                                                         NaN  \n",
       "1   0                                                         NaN  \n",
       "2   0                                                         NaN  \n",
       "    1                                  heart butterfly dear  5.02  \n",
       "3   0                                                         NaN  \n",
       "    1     someone like garden full flower need help tending  6.09  \n",
       "4   0                                                         NaN  \n",
       "    1                                             love help  5.27  \n",
       "    2                                                         NaN  \n",
       "5   0                                                         NaN  "
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# top 10 rows\n",
    "df_lines.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "47cd843d-d3b2-4378-b640-89d329c81d7c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculating age of character\n",
    "for character in df_characters['name']:\n",
    "    df_characters.loc[df_characters['name'] == character, 'age'] = df_lines[df_lines['character'] == character]['age'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "826a4624-ca35-4459-831f-3f6e06866329",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>num_dialogue</th>\n",
       "      <th>num_occurrence</th>\n",
       "      <th>age</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cid</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>the stranger</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>5.02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mrs. thompson</td>\n",
       "      <td>1</td>\n",
       "      <td>8</td>\n",
       "      <td>6.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>she</td>\n",
       "      <td>1</td>\n",
       "      <td>11</td>\n",
       "      <td>5.27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>that</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>24.66</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              name  num_dialogue  num_occurrence    age\n",
       "cid                                                    \n",
       "0     the stranger             1               1   5.02\n",
       "1    mrs. thompson             1               8   6.09\n",
       "2              she             1              11   5.27\n",
       "3             that             1               6  24.66"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# top 10 rows with age\n",
    "df_characters.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d79776c4-4d17-4494-93de-bc19ba57f504",
   "metadata": {},
   "source": [
    "# 5. Gender Detection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "c99a2192-044c-4f82-a19a-c07c120627c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# loading gender count vectorizer\n",
    "loaded_gender_cv = joblib.load('attributes/gender/gender_cv.joblib')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "f4153f64-b5cc-46c1-b5eb-8ec0fadeb2cd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# creating sparse matrix for transformed values and array\n",
    "gender_sparse_matrix = loaded_gender_cv.transform(df_lines['cleaned_dialogue'])\n",
    "gender_sparse_matrix_array = gender_sparse_matrix.toarray()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "50d4c7fc-c7c5-4236-beb8-e9a0c8d6d964",
   "metadata": {},
   "outputs": [],
   "source": [
    "# loading gender model\n",
    "loaded_gender_model = joblib.load('attributes/gender/gender_model.joblib')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "9d24b84d-7739-4f0f-9134-97c31329ab54",
   "metadata": {},
   "outputs": [],
   "source": [
    "# defining X and predicting\n",
    "df_lines['gender'] = loaded_gender_model.predict(gender_sparse_matrix_array)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "d5a1fa89-4d45-4f5f-99a6-7741e55792f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# define gender as None if dialogue is None\n",
    "df_lines.loc[df_lines['dialogue'].isna(), 'gender'] = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "9ea37aba-115f-46e2-95f7-816010b98327",
   "metadata": {},
   "outputs": [],
   "source": [
    "# define gender based upon he/she\n",
    "df_lines['gender'] = df_lines.apply(lambda row: 'Male' if row['character']=='he' else 'Female' if row['character']=='she' else row['gender'], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "dc5470bf-5ae6-4082-b10c-18cfb29b3dba",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>character</th>\n",
       "      <th>dialogue</th>\n",
       "      <th>narration</th>\n",
       "      <th>cleaned_dialogue</th>\n",
       "      <th>age</th>\n",
       "      <th>gender</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pid</th>\n",
       "      <th>ndid</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Once upon a time, in a small, quiet village, t...</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>One sunny afternoon, as Lily was by the river,...</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">2</th>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Unbeknownst to Lily, a kind stranger had been ...</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>the stranger</td>\n",
       "      <td>You have a heart as beautiful as that butterfl...</td>\n",
       "      <td>None</td>\n",
       "      <td>heart butterfly dear</td>\n",
       "      <td>5.02</td>\n",
       "      <td>Female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">3</th>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Lily blushed, not used to receiving compliment...</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mrs. thompson</td>\n",
       "      <td>I've been searching for someone just like you....</td>\n",
       "      <td>None</td>\n",
       "      <td>someone like garden full flower need help tending</td>\n",
       "      <td>6.09</td>\n",
       "      <td>Female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">4</th>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Lily's eyes sparkled with excitement. She had ...</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>she</td>\n",
       "      <td>I'd love to help,</td>\n",
       "      <td>None</td>\n",
       "      <td>love help</td>\n",
       "      <td>5.27</td>\n",
       "      <td>Female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>she replied.</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>From that day on, Lily spent her afternoons ca...</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              character                                           dialogue  \\\n",
       "pid ndid                                                                     \n",
       "0   0              None                                               None   \n",
       "1   0              None                                               None   \n",
       "2   0              None                                               None   \n",
       "    1      the stranger  You have a heart as beautiful as that butterfl...   \n",
       "3   0              None                                               None   \n",
       "    1     mrs. thompson  I've been searching for someone just like you....   \n",
       "4   0              None                                               None   \n",
       "    1               she                                  I'd love to help,   \n",
       "    2              None                                               None   \n",
       "5   0              None                                               None   \n",
       "\n",
       "                                                  narration  \\\n",
       "pid ndid                                                      \n",
       "0   0     Once upon a time, in a small, quiet village, t...   \n",
       "1   0     One sunny afternoon, as Lily was by the river,...   \n",
       "2   0     Unbeknownst to Lily, a kind stranger had been ...   \n",
       "    1                                                  None   \n",
       "3   0     Lily blushed, not used to receiving compliment...   \n",
       "    1                                                  None   \n",
       "4   0     Lily's eyes sparkled with excitement. She had ...   \n",
       "    1                                                  None   \n",
       "    2                                          she replied.   \n",
       "5   0     From that day on, Lily spent her afternoons ca...   \n",
       "\n",
       "                                           cleaned_dialogue   age  gender  \n",
       "pid ndid                                                                   \n",
       "0   0                                                         NaN    None  \n",
       "1   0                                                         NaN    None  \n",
       "2   0                                                         NaN    None  \n",
       "    1                                  heart butterfly dear  5.02  Female  \n",
       "3   0                                                         NaN    None  \n",
       "    1     someone like garden full flower need help tending  6.09  Female  \n",
       "4   0                                                         NaN    None  \n",
       "    1                                             love help  5.27  Female  \n",
       "    2                                                         NaN    None  \n",
       "5   0                                                         NaN    None  "
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# top 10 rows\n",
    "df_lines.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "0b4b3c81-57b1-42f8-8923-dd82e2ce1812",
   "metadata": {},
   "outputs": [],
   "source": [
    "# defining empty columns\n",
    "df_characters['male_count'], df_characters['female_count'] = None, None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "e85ee2b8-9774-4fec-8b3f-d45b1da2b907",
   "metadata": {},
   "outputs": [],
   "source": [
    "# counting male and female values of character\n",
    "for character in df_characters['name']:\n",
    "    df_characters.loc[df_characters['name'] == character, 'male_count'] = len(df_lines[(df_lines['character'] == character) & (df_lines['gender'] == \"Male\")])\n",
    "    df_characters.loc[df_characters['name'] == character, 'female_count'] = len(df_lines[(df_lines['character'] == character) & (df_lines['gender'] == \"Female\")])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "8cf24075-57fd-4a2c-9ae0-e3db7e23cda7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# defining gender column\n",
    "df_characters['gender'] = df_characters.apply(lambda row: 'Female' if row['female_count'] > row['male_count'] else ('Male' if row['female_count'] < row['male_count'] else None), axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "a4f85f64-9e64-44e4-bf6d-31c520fba523",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>num_dialogue</th>\n",
       "      <th>num_occurrence</th>\n",
       "      <th>age</th>\n",
       "      <th>male_count</th>\n",
       "      <th>female_count</th>\n",
       "      <th>gender</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cid</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>the stranger</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>5.02</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>Female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mrs. thompson</td>\n",
       "      <td>1</td>\n",
       "      <td>8</td>\n",
       "      <td>6.09</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>Female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>she</td>\n",
       "      <td>1</td>\n",
       "      <td>11</td>\n",
       "      <td>5.27</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>Female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>that</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>24.66</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>Male</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              name  num_dialogue  num_occurrence    age male_count  \\\n",
       "cid                                                                  \n",
       "0     the stranger             1               1   5.02          0   \n",
       "1    mrs. thompson             1               8   6.09          0   \n",
       "2              she             1              11   5.27          0   \n",
       "3             that             1               6  24.66          1   \n",
       "\n",
       "    female_count  gender  \n",
       "cid                       \n",
       "0              1  Female  \n",
       "1              1  Female  \n",
       "2              1  Female  \n",
       "3              0    Male  "
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# top 10 rows with male and female counts\n",
    "df_characters.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b0381a5-35a4-4f63-a637-046be8852e85",
   "metadata": {},
   "source": [
    "# 6. He/She Replacement"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "cd8b91f3-dded-46e0-9821-698047fa074b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['the stranger', 'mrs. thompson']\n"
     ]
    }
   ],
   "source": [
    "# characters not pronouns\n",
    "other_characters = df_characters[~df_characters['name'].isin(['he', 'she', 'that', 'they', 'it'])]['name']\n",
    "other_characters = list(other_characters)\n",
    "print(other_characters)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "758ba297-64ac-4247-9294-643fd3f98598",
   "metadata": {},
   "outputs": [],
   "source": [
    "# names in narration\n",
    "def find_names_in_narration(narration):\n",
    "    if narration is None:\n",
    "        return None\n",
    "    names = dict()\n",
    "    for character in other_characters:\n",
    "        location = narration.lower().rfind(character)\n",
    "        if location != -1:\n",
    "            names[character] = location\n",
    "    if len(names) == 0:\n",
    "        return None\n",
    "    return names.keys()\n",
    "df_lines['narrated_names'] = df_lines['narration'].apply(lambda row: find_names_in_narration(row))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "d7ba19c8-0b9c-43a6-84b6-2869a87e36da",
   "metadata": {},
   "outputs": [],
   "source": [
    "# he she replacement by names\n",
    "def he_she_replacement(i, gender):\n",
    "    j = i-1\n",
    "    last_row_narrated_names = df_lines.iloc[j]['narrated_names']\n",
    "    while last_row_narrated_names is None:\n",
    "        j -= 1\n",
    "        last_row_narrated_names = df_lines.iloc[j]['narrated_names']\n",
    "    last_row_narrated_names = list(last_row_narrated_names)\n",
    "    for name in last_row_narrated_names:\n",
    "        if list(df_characters[df_characters['name'] == name]['gender'])[0] == gender:\n",
    "            return name\n",
    "    return last_row_narrated_names[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "e7a2b3d4-e84d-4a4e-b61e-535de249afba",
   "metadata": {},
   "outputs": [],
   "source": [
    "# replacing all he/she in df_lines\n",
    "for i in range(1, len(df_lines)):\n",
    "    if df_lines.iloc[i, 0] not in other_characters+['he', 'she'] and df_lines.iloc[i, 0] is not None:\n",
    "        df_lines.iloc[i, 0] = None\n",
    "        df_lines.iloc[i, 4] = None\n",
    "    elif df_lines.iloc[i, 0] == 'he':\n",
    "        df_lines.iloc[i, 0] = he_she_replacement(i, 'Male')\n",
    "    elif df_lines.iloc[i, 0] == 'she':\n",
    "        df_lines.iloc[i, 0] = he_she_replacement(i, 'Female')\n",
    "    else:\n",
    "        pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "c178e42e-5ee8-48c5-b342-9fb8a30e2901",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>character</th>\n",
       "      <th>dialogue</th>\n",
       "      <th>narration</th>\n",
       "      <th>cleaned_dialogue</th>\n",
       "      <th>age</th>\n",
       "      <th>gender</th>\n",
       "      <th>narrated_names</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pid</th>\n",
       "      <th>ndid</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">4</th>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Lily's eyes sparkled with excitement. She had ...</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>(mrs. thompson)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mrs. thompson</td>\n",
       "      <td>I'd love to help,</td>\n",
       "      <td>None</td>\n",
       "      <td>love help</td>\n",
       "      <td>5.27</td>\n",
       "      <td>Female</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>she replied.</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>From that day on, Lily spent her afternoons ca...</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>(mrs. thompson)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Lily and Mrs. Thompson developed a deep and he...</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>(mrs. thompson)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>As the years went by, Lily continued to visit ...</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>(mrs. thompson)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">8</th>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>One fateful day, Mrs. Thompson passed away, le...</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>(mrs. thompson)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>None</td>\n",
       "      <td>May the kindness you showed to a struggling bu...</td>\n",
       "      <td>None</td>\n",
       "      <td>may kindness butterfly world beauty flower</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Male</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Lily kept her promise, and the garden continue...</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>(mrs. thompson)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>And so, in the memory of a chance encounter an...</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              character                                           dialogue  \\\n",
       "pid ndid                                                                     \n",
       "4   0              None                                               None   \n",
       "    1     mrs. thompson                                  I'd love to help,   \n",
       "    2              None                                               None   \n",
       "5   0              None                                               None   \n",
       "6   0              None                                               None   \n",
       "7   0              None                                               None   \n",
       "8   0              None                                               None   \n",
       "    1              None  May the kindness you showed to a struggling bu...   \n",
       "9   0              None                                               None   \n",
       "10  0              None                                               None   \n",
       "\n",
       "                                                  narration  \\\n",
       "pid ndid                                                      \n",
       "4   0     Lily's eyes sparkled with excitement. She had ...   \n",
       "    1                                                  None   \n",
       "    2                                          she replied.   \n",
       "5   0     From that day on, Lily spent her afternoons ca...   \n",
       "6   0     Lily and Mrs. Thompson developed a deep and he...   \n",
       "7   0     As the years went by, Lily continued to visit ...   \n",
       "8   0     One fateful day, Mrs. Thompson passed away, le...   \n",
       "    1                                                  None   \n",
       "9   0     Lily kept her promise, and the garden continue...   \n",
       "10  0     And so, in the memory of a chance encounter an...   \n",
       "\n",
       "                                    cleaned_dialogue   age  gender  \\\n",
       "pid ndid                                                             \n",
       "4   0                                                  NaN    None   \n",
       "    1                                      love help  5.27  Female   \n",
       "    2                                                  NaN    None   \n",
       "5   0                                                  NaN    None   \n",
       "6   0                                                  NaN    None   \n",
       "7   0                                                  NaN    None   \n",
       "8   0                                                  NaN    None   \n",
       "    1     may kindness butterfly world beauty flower   NaN    Male   \n",
       "9   0                                                  NaN    None   \n",
       "10  0                                                  NaN    None   \n",
       "\n",
       "           narrated_names  \n",
       "pid ndid                   \n",
       "4   0     (mrs. thompson)  \n",
       "    1                None  \n",
       "    2                None  \n",
       "5   0     (mrs. thompson)  \n",
       "6   0     (mrs. thompson)  \n",
       "7   0     (mrs. thompson)  \n",
       "8   0     (mrs. thompson)  \n",
       "    1                None  \n",
       "9   0     (mrs. thompson)  \n",
       "10  0                None  "
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# lines with narrated names\n",
    "df_lines.tail(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "38e29a0e-d927-4080-8ff7-3a41aa4a8948",
   "metadata": {},
   "outputs": [],
   "source": [
    "# filtered df_characters with other_characters\n",
    "df_characters = df_characters[df_characters['name'].isin(other_characters)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "f4069438-e06e-461e-93bc-df943aa9aca6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>num_dialogue</th>\n",
       "      <th>num_occurrence</th>\n",
       "      <th>age</th>\n",
       "      <th>male_count</th>\n",
       "      <th>female_count</th>\n",
       "      <th>gender</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cid</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>the stranger</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>5.02</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>Female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mrs. thompson</td>\n",
       "      <td>1</td>\n",
       "      <td>8</td>\n",
       "      <td>6.09</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>Female</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              name  num_dialogue  num_occurrence   age male_count  \\\n",
       "cid                                                                 \n",
       "0     the stranger             1               1  5.02          0   \n",
       "1    mrs. thompson             1               8  6.09          0   \n",
       "\n",
       "    female_count  gender  \n",
       "cid                       \n",
       "0              1  Female  \n",
       "1              1  Female  "
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# filtered df_characters only\n",
    "df_characters.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "id": "029a0ce5-b696-4a39-a158-0bb5ab5e2436",
   "metadata": {},
   "outputs": [],
   "source": [
    "# gathering characters from df_lines\n",
    "df_lines_vc = df_lines['character'].value_counts()\n",
    "filtered_df = pd.DataFrame(df_lines_vc.reset_index(), columns=['character', 'count'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "id": "5d398532-3b4f-4df0-82ae-55e1196fbbee",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>character</th>\n",
       "      <th>count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>mrs. thompson</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>the stranger</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       character  count\n",
       "0  mrs. thompson      2\n",
       "1   the stranger      1"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# filtered_df\n",
    "filtered_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "id": "b904bd8b-999e-44b9-b627-ae0ee93de8ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "# number of dialogues, male and female counts and gender after he/she replacement\n",
    "df_characters['num_dialogue'] = df_characters['name'].apply(lambda x: list(filtered_df[filtered_df['character'] == x]['count'])[0])\n",
    "\n",
    "for character in df_characters['name']:\n",
    "    df_characters.loc[df_characters['name'] == character, 'male_count'] = len(df_lines[(df_lines['character'] == character) & (df_lines['gender'] == \"Male\")])\n",
    "    df_characters.loc[df_characters['name'] == character, 'female_count'] = len(df_lines[(df_lines['character'] == character) & (df_lines['gender'] == \"Female\")])\n",
    "\n",
    "df_characters['gender'] = df_characters.apply(lambda row: 'Female' if row['female_count'] > row['male_count'] else ('Male' if row['female_count'] < row['male_count'] else None), axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "id": "cfbc407c-b3f9-44c8-b779-e02f54bd46e7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>num_dialogue</th>\n",
       "      <th>num_occurrence</th>\n",
       "      <th>age</th>\n",
       "      <th>male_count</th>\n",
       "      <th>female_count</th>\n",
       "      <th>gender</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cid</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>the stranger</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>5.02</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>Female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mrs. thompson</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>6.09</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>Female</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              name  num_dialogue  num_occurrence   age male_count  \\\n",
       "cid                                                                 \n",
       "0     the stranger             1               1  5.02          0   \n",
       "1    mrs. thompson             2               8  6.09          0   \n",
       "\n",
       "    female_count  gender  \n",
       "cid                       \n",
       "0              1  Female  \n",
       "1              2  Female  "
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# new df_characters with new number of dialogues, male count and female count\n",
    "df_characters.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4495c2b3-d9d0-40e6-80be-5d4626cea903",
   "metadata": {},
   "source": [
    "# 7. Age and Gender Editing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "id": "a1c5e0dd-518a-4d76-bb8d-e772de6afd25",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "THE STRANGER\n",
      "Age predicted:40.0\n"
     ]
    },
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      "Enter age (or leave blank if you are ok with predicted age.) 40\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Gender predicted:Female\n"
     ]
    },
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      "Enter gender (Male/Female/None or leave blank if you are ok with predicted gender.) Female\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "======================================\n",
      "MRS. THOMPSON\n",
      "Age predicted:40.0\n"
     ]
    },
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      "Enter age (or leave blank if you are ok with predicted age.) 40\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Gender predicted:Female\n"
     ]
    },
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      "Enter gender (Male/Female/None or leave blank if you are ok with predicted gender.) Female\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "======================================\n"
     ]
    }
   ],
   "source": [
    "# inputs for new age and gender\n",
    "for index, row in df_characters.iterrows():\n",
    "    print(row['name'].upper())\n",
    "    print(f\"Age predicted:{row['age']}\")\n",
    "    new_age = input(\"Enter age (or leave blank if you are ok with predicted age.)\")\n",
    "    df_characters.loc[df_characters['name'] == row['name'], 'age'] = float(new_age) if new_age != '' else row['age']\n",
    "    print(f\"Gender predicted:{row['gender']}\")\n",
    "    new_gender = input(\"Enter gender (Male/Female/None or leave blank if you are ok with predicted gender.)\")\n",
    "    df_characters.loc[df_characters['name'] == row['name'], 'gender'] = new_gender if new_gender != '' else row['gender']\n",
    "    print(\"======================================\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "id": "7903cfc1-837a-4b51-9184-53db0efbd583",
   "metadata": {},
   "outputs": [],
   "source": [
    "# replace 'None' by None\n",
    "df_characters[df_characters['gender'] == 'None'] = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "id": "7274707a-ef84-4371-94ec-266c245d7f83",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([40.])"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# checking ages\n",
    "df_characters['age'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "id": "d9d6958e-bb8a-46fb-8388-7d02963d23a6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Female'], dtype=object)"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# checking genders\n",
    "df_characters['gender'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "id": "e03c9259-cb3f-4d3b-9d5b-ac3fd848e92f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>num_dialogue</th>\n",
       "      <th>num_occurrence</th>\n",
       "      <th>age</th>\n",
       "      <th>male_count</th>\n",
       "      <th>female_count</th>\n",
       "      <th>gender</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cid</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>the stranger</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>40.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>Female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mrs. thompson</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>40.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>Female</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              name  num_dialogue  num_occurrence   age male_count  \\\n",
       "cid                                                                 \n",
       "0     the stranger             1               1  40.0          0   \n",
       "1    mrs. thompson             2               8  40.0          0   \n",
       "\n",
       "    female_count  gender  \n",
       "cid                       \n",
       "0              1  Female  \n",
       "1              2  Female  "
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# checking characters\n",
    "df_characters.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22e4292b-e0cf-473e-a67f-bec413f76f2b",
   "metadata": {},
   "source": [
    "# 8. Gender-based Voice Selection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "id": "ff5f075a-f034-4629-bcb3-cdffa986761c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create voice engine for voice selection\n",
    "class VoiceEngine:\n",
    "\n",
    "    def __init__(self, character):\n",
    "        self.engine = pyttsx3.init()\n",
    "        voices = self.engine.getProperty('voices')\n",
    "        speakers = {\n",
    "            'David': voices[0],\n",
    "            'James': voices[1],\n",
    "            'Linda': voices[2],\n",
    "            'Richard': voices[3],\n",
    "            'George': voices[4],\n",
    "            'Susan': voices[5],\n",
    "            'Sean': voices[6],\n",
    "            'Heera': voices[7],\n",
    "            'Ravi': voices[8],\n",
    "            'Mark': voices[9],\n",
    "            'Hazel': voices[10],\n",
    "            'Catherine': voices[11],\n",
    "            'Zira': voices[12]\n",
    "        }\n",
    "        self.engine.setProperty('voice', speakers[character].id)\n",
    "\n",
    "    def dialogue_delivery(self, dialogue):\n",
    "        self.engine.say(dialogue)\n",
    "        self.engine.runAndWait()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "id": "f127c404-8a38-4077-936d-3b4b77b04fcb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# definining characters\n",
    "males = ['David', 'James', 'Richard', 'George', 'Sean', 'Ravi', 'Mark']\n",
    "females = ['Linda', 'Susan', 'Heera', 'Hazel', 'Catherine', 'Zira']\n",
    "\n",
    "# unused characters\n",
    "unused_males = males.copy()\n",
    "unused_females = females.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "id": "2d0fbe5e-d5eb-4c60-a138-ebc054e9a59b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# empty column\n",
    "df_characters['voice'] = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "id": "17a6b9f6-14f8-4643-a857-3a26e331dee5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# allocating voice to all characters\n",
    "for index, row in df_characters.iterrows():\n",
    "    if row['gender'] == 'Male':\n",
    "        random_number = random.randint(0, len(unused_males)-1)\n",
    "        name_of_voice = unused_males[random_number]\n",
    "        df_characters.loc[index, 'voice'] = name_of_voice\n",
    "        unused_males.remove(name_of_voice)\n",
    "        if len(unused_males) == 0:\n",
    "            unused_males = males.copy()\n",
    "    elif row['gender'] == 'Female':\n",
    "        random_number = random.randint(0, len(unused_females)-1)\n",
    "        name_of_voice = unused_females[random_number]\n",
    "        df_characters.loc[index, 'voice'] = name_of_voice\n",
    "        unused_females.remove(name_of_voice)\n",
    "        if len(unused_females) == 0:\n",
    "            unused_females = females.copy()\n",
    "    else:\n",
    "        unused_all = unused_males + unused_females\n",
    "        random_number = random.randint(0, len(unused_all)-1)\n",
    "        name_of_voice = unused_all[random_number]\n",
    "        df_characters.loc[index, 'voice'] = name_of_voice\n",
    "        if name_of_voice in unused_males:\n",
    "            unused_males.remove(name_of_voice)\n",
    "            if len(unused_males) == 0:\n",
    "                unused_males = males.copy()\n",
    "        else:\n",
    "            unused_females.remove(name_of_voice)\n",
    "            if len(unused_females) == 0:\n",
    "                unused_females = females.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "id": "c33bc97d-d78b-431f-819b-77f4851d7c92",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>num_dialogue</th>\n",
       "      <th>num_occurrence</th>\n",
       "      <th>age</th>\n",
       "      <th>male_count</th>\n",
       "      <th>female_count</th>\n",
       "      <th>gender</th>\n",
       "      <th>voice</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cid</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>the stranger</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>40.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>Female</td>\n",
       "      <td>Linda</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mrs. thompson</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>40.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>Female</td>\n",
       "      <td>Heera</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              name  num_dialogue  num_occurrence   age male_count  \\\n",
       "cid                                                                 \n",
       "0     the stranger             1               1  40.0          0   \n",
       "1    mrs. thompson             2               8  40.0          0   \n",
       "\n",
       "    female_count  gender  voice  \n",
       "cid                              \n",
       "0              1  Female  Linda  \n",
       "1              2  Female  Heera  "
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# checking characters with voice\n",
    "df_characters.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "id": "b3dfb8fa-47ae-400c-8d7b-df8a022f5360",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['David', 'James', 'Richard', 'George', 'Sean', 'Ravi', 'Mark'] ['Susan', 'Hazel', 'Catherine', 'Zira']\n"
     ]
    }
   ],
   "source": [
    "# checking remaining voices\n",
    "print(unused_males, unused_females)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "id": "47be7b03-c233-4d9c-829b-63852097aa02",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Narrator voice: Zira\n"
     ]
    }
   ],
   "source": [
    "# assigning voice to narrator\n",
    "unused_all = unused_males + unused_females\n",
    "random_number = random.randint(0, len(unused_all)-1)\n",
    "narrator_voice = unused_all[random_number]\n",
    "print(f\"Narrator voice: {narrator_voice}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "481d1aaa-e9a5-41c2-b616-ca61b59102d0",
   "metadata": {},
   "source": [
    "# 8. Pitch-Tempo Defining by Age"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "id": "3df2f883-56cf-49a7-876e-7abe7460f8a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# pitch estimation in Hz and tempo estimation in words per minute by age\n",
    "def pitch_and_tempo_estimation(age):\n",
    "    if age <= 5:\n",
    "        result = [300, 150]\n",
    "    elif age > 5 and age <= 12:\n",
    "        result = [250, 160]\n",
    "    elif age > 12 and age <= 18:\n",
    "        result = [200, 170]\n",
    "    elif age > 18 and age <= 30:\n",
    "        result = [180, 180]\n",
    "    elif age > 30 and age <= 40:\n",
    "        result = [160, 175]\n",
    "    elif age > 40 and age <= 60:\n",
    "        result = [150, 160]\n",
    "    else:\n",
    "        result = [130, 150]\n",
    "    return pd.Series(result, index=('pitch', 'tempo'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "id": "d6b176f6-d8ee-4238-b291-94ca9d41e8e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# adding pitch and tempo columns by characters' age\n",
    "df_characters[['pitch', 'tempo']] = df_characters['age'].apply(pitch_and_tempo_estimation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "id": "5adaa2c1-c53c-4c98-9fa2-fd30e772a846",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>num_dialogue</th>\n",
       "      <th>num_occurrence</th>\n",
       "      <th>age</th>\n",
       "      <th>male_count</th>\n",
       "      <th>female_count</th>\n",
       "      <th>gender</th>\n",
       "      <th>voice</th>\n",
       "      <th>pitch</th>\n",
       "      <th>tempo</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cid</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>the stranger</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>40.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>Female</td>\n",
       "      <td>Linda</td>\n",
       "      <td>160</td>\n",
       "      <td>175</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mrs. thompson</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>40.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>Female</td>\n",
       "      <td>Heera</td>\n",
       "      <td>160</td>\n",
       "      <td>175</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              name  num_dialogue  num_occurrence   age male_count  \\\n",
       "cid                                                                 \n",
       "0     the stranger             1               1  40.0          0   \n",
       "1    mrs. thompson             2               8  40.0          0   \n",
       "\n",
       "    female_count  gender  voice  pitch  tempo  \n",
       "cid                                            \n",
       "0              1  Female  Linda    160    175  \n",
       "1              2  Female  Heera    160    175  "
      ]
     },
     "execution_count": 137,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# checking table with pitch, tempo and voice\n",
    "df_characters.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48c33ff0-4636-4dc5-afd1-54b2eff88455",
   "metadata": {},
   "source": [
    "# 9. Emotions Detection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "id": "e6686fbb-a7a8-492e-8b0e-9cabe86c5220",
   "metadata": {},
   "outputs": [],
   "source": [
    "# creating emotion detection classifier\n",
    "ed_classifier = pipeline(\"text-classification\", model=\"j-hartmann/emotion-english-distilroberta-base\", return_all_scores=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "id": "6c89b0cd-ba34-4cd0-a761-c52dd210dadd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# function to return better dictionary\n",
    "def show_emotions(dialogue):\n",
    "    emotions_dict = dict()\n",
    "    edc = ed_classifier(dialogue)\n",
    "    for x in edc[0]:\n",
    "        emotions_dict[x['label']] = x['score']\n",
    "    return emotions_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "id": "184ea366-fe55-4205-9be0-86b291e44d54",
   "metadata": {},
   "outputs": [],
   "source": [
    "# using ed_classifier for showing emotions for each dialogue\n",
    "df_lines['anger'], df_lines['disgust'], df_lines['fear'], df_lines['joy'], df_lines['neutral'], df_lines['sadness'], df_lines['surprise'] = None, None, None, None, None, None, None\n",
    "df_lines[['anger', 'disgust', 'fear', 'joy', 'neutral', 'sadness', 'surprise']] = df_lines['dialogue'].apply(lambda x: list(show_emotions(x).values()) if x is not None else [None, None, None, None, None, None, None]).tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "id": "138d46ee-882d-429d-8b30-3651a0796574",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>character</th>\n",
       "      <th>dialogue</th>\n",
       "      <th>narration</th>\n",
       "      <th>cleaned_dialogue</th>\n",
       "      <th>age</th>\n",
       "      <th>gender</th>\n",
       "      <th>narrated_names</th>\n",
       "      <th>anger</th>\n",
       "      <th>disgust</th>\n",
       "      <th>fear</th>\n",
       "      <th>joy</th>\n",
       "      <th>neutral</th>\n",
       "      <th>sadness</th>\n",
       "      <th>surprise</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pid</th>\n",
       "      <th>ndid</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Midnight in the heart of the city, the neon si...</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>(alex)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>The door creaked open, and a mysterious figure...</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>(alex)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">2</th>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Alex raised an eyebrow.</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>(alex)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>alex</td>\n",
       "      <td>Late night for a coffee, isn't it?</td>\n",
       "      <td>None</td>\n",
       "      <td>night coffee</td>\n",
       "      <td>20.47</td>\n",
       "      <td>Female</td>\n",
       "      <td>None</td>\n",
       "      <td>0.061734</td>\n",
       "      <td>0.377704</td>\n",
       "      <td>0.022014</td>\n",
       "      <td>0.010213</td>\n",
       "      <td>0.325844</td>\n",
       "      <td>0.041864</td>\n",
       "      <td>0.160626</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">3</th>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>The stranger chuckled, a voice tinged with int...</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>(the stranger)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>the stranger</td>\n",
       "      <td>Some stories unfold when the world sleeps.</td>\n",
       "      <td>None</td>\n",
       "      <td>story world sleep</td>\n",
       "      <td>14.27</td>\n",
       "      <td>Male</td>\n",
       "      <td>None</td>\n",
       "      <td>0.015564</td>\n",
       "      <td>0.095981</td>\n",
       "      <td>0.034734</td>\n",
       "      <td>0.003982</td>\n",
       "      <td>0.776333</td>\n",
       "      <td>0.044221</td>\n",
       "      <td>0.029184</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">4</th>\n",
       "      <th>0</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Intrigued, Alex leaned forward.</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>(alex)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>alex</td>\n",
       "      <td>What kind of stories?</td>\n",
       "      <td>None</td>\n",
       "      <td>kind story</td>\n",
       "      <td>5.31</td>\n",
       "      <td>Male</td>\n",
       "      <td>None</td>\n",
       "      <td>0.015280</td>\n",
       "      <td>0.010232</td>\n",
       "      <td>0.004569</td>\n",
       "      <td>0.004578</td>\n",
       "      <td>0.831131</td>\n",
       "      <td>0.003179</td>\n",
       "      <td>0.131032</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">5</th>\n",
       "      <th>0</th>\n",
       "      <td>the stranger</td>\n",
       "      <td>Stories of the forgotten, the ones that only e...</td>\n",
       "      <td>None</td>\n",
       "      <td>story one city silence</td>\n",
       "      <td>5.31</td>\n",
       "      <td>Male</td>\n",
       "      <td>None</td>\n",
       "      <td>0.010006</td>\n",
       "      <td>0.046537</td>\n",
       "      <td>0.044176</td>\n",
       "      <td>0.002887</td>\n",
       "      <td>0.560465</td>\n",
       "      <td>0.322675</td>\n",
       "      <td>0.013254</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>the stranger replied mysteriously.</td>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>(the stranger)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             character                                           dialogue  \\\n",
       "pid ndid                                                                    \n",
       "0   0             None                                               None   \n",
       "1   0             None                                               None   \n",
       "2   0             None                                               None   \n",
       "    1             alex                 Late night for a coffee, isn't it?   \n",
       "3   0             None                                               None   \n",
       "    1     the stranger         Some stories unfold when the world sleeps.   \n",
       "4   0             None                                               None   \n",
       "    1             alex                              What kind of stories?   \n",
       "5   0     the stranger  Stories of the forgotten, the ones that only e...   \n",
       "    1             None                                               None   \n",
       "\n",
       "                                                  narration  \\\n",
       "pid ndid                                                      \n",
       "0   0     Midnight in the heart of the city, the neon si...   \n",
       "1   0     The door creaked open, and a mysterious figure...   \n",
       "2   0                              Alex raised an eyebrow.    \n",
       "    1                                                  None   \n",
       "3   0     The stranger chuckled, a voice tinged with int...   \n",
       "    1                                                  None   \n",
       "4   0                      Intrigued, Alex leaned forward.    \n",
       "    1                                                  None   \n",
       "5   0                                                  None   \n",
       "    1                    the stranger replied mysteriously.   \n",
       "\n",
       "                cleaned_dialogue    age  gender  narrated_names     anger  \\\n",
       "pid ndid                                                                    \n",
       "0   0                               NaN    None          (alex)       NaN   \n",
       "1   0                               NaN    None          (alex)       NaN   \n",
       "2   0                               NaN    None          (alex)       NaN   \n",
       "    1               night coffee  20.47  Female            None  0.061734   \n",
       "3   0                               NaN    None  (the stranger)       NaN   \n",
       "    1          story world sleep  14.27    Male            None  0.015564   \n",
       "4   0                               NaN    None          (alex)       NaN   \n",
       "    1                 kind story   5.31    Male            None  0.015280   \n",
       "5   0     story one city silence   5.31    Male            None  0.010006   \n",
       "    1                               NaN    None  (the stranger)       NaN   \n",
       "\n",
       "           disgust      fear       joy   neutral   sadness  surprise  \n",
       "pid ndid                                                              \n",
       "0   0          NaN       NaN       NaN       NaN       NaN       NaN  \n",
       "1   0          NaN       NaN       NaN       NaN       NaN       NaN  \n",
       "2   0          NaN       NaN       NaN       NaN       NaN       NaN  \n",
       "    1     0.377704  0.022014  0.010213  0.325844  0.041864  0.160626  \n",
       "3   0          NaN       NaN       NaN       NaN       NaN       NaN  \n",
       "    1     0.095981  0.034734  0.003982  0.776333  0.044221  0.029184  \n",
       "4   0          NaN       NaN       NaN       NaN       NaN       NaN  \n",
       "    1     0.010232  0.004569  0.004578  0.831131  0.003179  0.131032  \n",
       "5   0     0.046537  0.044176  0.002887  0.560465  0.322675  0.013254  \n",
       "    1          NaN       NaN       NaN       NaN       NaN       NaN  "
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# top 10 rows\n",
    "df_lines.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "id": "3b920107-822b-40ac-ac94-d7b778a6f866",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>dialogue</th>\n",
       "      <th>anger</th>\n",
       "      <th>disgust</th>\n",
       "      <th>fear</th>\n",
       "      <th>joy</th>\n",
       "      <th>neutral</th>\n",
       "      <th>sadness</th>\n",
       "      <th>surprise</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pid</th>\n",
       "      <th>ndid</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <th>1</th>\n",
       "      <td>Late night for a coffee, isn't it?</td>\n",
       "      <td>0.061734</td>\n",
       "      <td>0.377704</td>\n",
       "      <td>0.022014</td>\n",
       "      <td>0.010213</td>\n",
       "      <td>0.325844</td>\n",
       "      <td>0.041864</td>\n",
       "      <td>0.160626</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <th>1</th>\n",
       "      <td>Some stories unfold when the world sleeps.</td>\n",
       "      <td>0.015564</td>\n",
       "      <td>0.095981</td>\n",
       "      <td>0.034734</td>\n",
       "      <td>0.003982</td>\n",
       "      <td>0.776333</td>\n",
       "      <td>0.044221</td>\n",
       "      <td>0.029184</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <th>1</th>\n",
       "      <td>What kind of stories?</td>\n",
       "      <td>0.015280</td>\n",
       "      <td>0.010232</td>\n",
       "      <td>0.004569</td>\n",
       "      <td>0.004578</td>\n",
       "      <td>0.831131</td>\n",
       "      <td>0.003179</td>\n",
       "      <td>0.131032</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <th>0</th>\n",
       "      <td>Stories of the forgotten, the ones that only e...</td>\n",
       "      <td>0.010006</td>\n",
       "      <td>0.046537</td>\n",
       "      <td>0.044176</td>\n",
       "      <td>0.002887</td>\n",
       "      <td>0.560465</td>\n",
       "      <td>0.322675</td>\n",
       "      <td>0.013254</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">7</th>\n",
       "      <th>0</th>\n",
       "      <td>You know,</td>\n",
       "      <td>0.076726</td>\n",
       "      <td>0.240470</td>\n",
       "      <td>0.025607</td>\n",
       "      <td>0.024524</td>\n",
       "      <td>0.607015</td>\n",
       "      <td>0.013988</td>\n",
       "      <td>0.011670</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>this feels like a chapter from a novel. A meet...</td>\n",
       "      <td>0.011276</td>\n",
       "      <td>0.111364</td>\n",
       "      <td>0.222619</td>\n",
       "      <td>0.005216</td>\n",
       "      <td>0.544812</td>\n",
       "      <td>0.012213</td>\n",
       "      <td>0.092499</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <th>1</th>\n",
       "      <td>Perhaps it is. Life has a way of scripting its...</td>\n",
       "      <td>0.005774</td>\n",
       "      <td>0.010377</td>\n",
       "      <td>0.002170</td>\n",
       "      <td>0.002140</td>\n",
       "      <td>0.962292</td>\n",
       "      <td>0.004725</td>\n",
       "      <td>0.012522</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <th>1</th>\n",
       "      <td>Until our paths cross again, Alex.</td>\n",
       "      <td>0.015226</td>\n",
       "      <td>0.018030</td>\n",
       "      <td>0.068527</td>\n",
       "      <td>0.014377</td>\n",
       "      <td>0.815693</td>\n",
       "      <td>0.064210</td>\n",
       "      <td>0.003937</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   dialogue     anger  \\\n",
       "pid ndid                                                                \n",
       "2   1                    Late night for a coffee, isn't it?  0.061734   \n",
       "3   1            Some stories unfold when the world sleeps.  0.015564   \n",
       "4   1                                 What kind of stories?  0.015280   \n",
       "5   0     Stories of the forgotten, the ones that only e...  0.010006   \n",
       "7   0                                             You know,  0.076726   \n",
       "    2     this feels like a chapter from a novel. A meet...  0.011276   \n",
       "8   1     Perhaps it is. Life has a way of scripting its...  0.005774   \n",
       "9   1                    Until our paths cross again, Alex.  0.015226   \n",
       "\n",
       "           disgust      fear       joy   neutral   sadness  surprise  \n",
       "pid ndid                                                              \n",
       "2   1     0.377704  0.022014  0.010213  0.325844  0.041864  0.160626  \n",
       "3   1     0.095981  0.034734  0.003982  0.776333  0.044221  0.029184  \n",
       "4   1     0.010232  0.004569  0.004578  0.831131  0.003179  0.131032  \n",
       "5   0     0.046537  0.044176  0.002887  0.560465  0.322675  0.013254  \n",
       "7   0     0.240470  0.025607  0.024524  0.607015  0.013988  0.011670  \n",
       "    2     0.111364  0.222619  0.005216  0.544812  0.012213  0.092499  \n",
       "8   1     0.010377  0.002170  0.002140  0.962292  0.004725  0.012522  \n",
       "9   1     0.018030  0.068527  0.014377  0.815693  0.064210  0.003937  "
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check dialogues and corresponding emotions only\n",
    "df_lines[~df_lines['dialogue'].isna()][['dialogue', 'anger', 'disgust', 'fear', 'joy', 'neutral', 'sadness', 'surprise']].head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86f5a68c-2349-485e-91e1-97cd60921aa7",
   "metadata": {},
   "source": [
    "# 10. Audio Generation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "id": "2293d368-4d4a-4613-bdf8-64a3b8e6854d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step 1: CONVERSIONS\n",
      "Line 1/19 converted.\n",
      "Line 2/19 converted.\n",
      "Line 3/19 converted.\n",
      "Line 4/19 converted.\n",
      "Line 5/19 converted.\n",
      "Line 6/19 converted.\n",
      "Line 7/19 converted.\n",
      "Line 8/19 converted.\n",
      "Line 9/19 converted.\n",
      "Line 10/19 converted.\n",
      "Line 11/19 converted.\n",
      "Line 12/19 converted.\n",
      "Line 13/19 converted.\n",
      "Line 14/19 converted.\n",
      "Line 15/19 converted.\n",
      "Line 16/19 converted.\n",
      "Line 17/19 converted.\n",
      "Line 18/19 converted.\n",
      "Line 19/19 converted.\n",
      "Conversions finished.\n"
     ]
    }
   ],
   "source": [
    "# creating audios\n",
    "print(\"Step 1: CONVERSIONS\")\n",
    "i = 0\n",
    "for index, row in df_lines.iterrows():\n",
    "    if row['dialogue'] is not None:\n",
    "        speech_gtts = gTTS(text=row['dialogue'], lang='en', slow=False, tld='co.in')\n",
    "    else:\n",
    "        speech_gtts = gTTS(text=row['narration'], lang='en', slow=False, tld='us')\n",
    "    temp_file = f\"conversions/{i}.mp3\"\n",
    "    print(f\"Line {i+1}/{df_lines.shape[0]} converted.\")\n",
    "    speech_gtts.save(temp_file)\n",
    "    i += 1\n",
    "print(\"Conversions finished.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "id": "cb7e927d-6822-4398-8733-4ef1a04c625b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step 2: COMBINATION\n",
      "Combination finished.\n"
     ]
    }
   ],
   "source": [
    "# combining audios\n",
    "print(\"Step 2: COMBINATION\")\n",
    "clips = [AudioFileClip(f\"conversions/{i}.mp3\") for i in range(df_lines.shape[0])]\n",
    "final_clip = concatenate_audioclips(clips)\n",
    "print(\"Combination finished.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "id": "a41b7618-bf69-4599-afef-27abc652c41f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step 3: ADJUSTMENT\n",
      "MoviePy - Writing audio in conversions/final_slow.mp3\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                                                       \r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MoviePy - Done.\n",
      "final_slow.mp3 downloaded.\n",
      "Speeded over.\n"
     ]
    }
   ],
   "source": [
    "# adjust speed\n",
    "print(\"Step 3: ADJUSTMENT\")\n",
    "final_clip.write_audiofile(\"conversions/final_slow.mp3\")\n",
    "print(\"final_slow.mp3 downloaded.\")\n",
    "y, sr = librosa.load(\"conversions/final_slow.mp3\", sr=None)\n",
    "y_speed = librosa.effects.time_stretch(y, rate=1.25)\n",
    "print(\"Speeded over.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "id": "664227b9-240e-41cf-95ac-832ca3b91ca7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step 4: REMOVAL\n",
      "Removed 0.mp3\n",
      "Removed 1.mp3\n",
      "Removed 2.mp3\n",
      "Removed 3.mp3\n",
      "Removed 4.mp3\n",
      "Removed 5.mp3\n",
      "Removed 6.mp3\n",
      "Removed 7.mp3\n",
      "Removed 8.mp3\n",
      "Removed 9.mp3\n",
      "Removed 10.mp3\n",
      "Removed 11.mp3\n",
      "Removed 12.mp3\n",
      "Removed 13.mp3\n",
      "Removed 14.mp3\n",
      "Removed 15.mp3\n",
      "Removed 16.mp3\n",
      "Removed 17.mp3\n",
      "Removed 18.mp3\n",
      "Removal over.\n"
     ]
    }
   ],
   "source": [
    "# removing audios\n",
    "print(\"Step 4: REMOVAL\")\n",
    "for i in range(df_lines.shape[0]):\n",
    "    os.remove(f\"conversions/{i}.mp3\")\n",
    "    print(f\"Removed {i}.mp3\")\n",
    "os.remove(\"conversions/final_slow.mp3\")\n",
    "print(\"Removal over.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "id": "665dd587-e7b8-4d10-beeb-41a3d3867374",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step 5: DOWNLOADING\n",
      "cafe_at_midnight.mp3 downloaded\n"
     ]
    }
   ],
   "source": [
    "# downloading final audio\n",
    "print(\"Step 5: DOWNLOADING\")\n",
    "final_title = title.replace(\" \", \"_\")\n",
    "sf.write(f\"audiobooks/{final_title}.mp3\", y_speed, sr)\n",
    "print(f\"{final_title}.mp3 downloaded\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b793686a-0276-48fb-be53-3ccd40ff0675",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}