Generative_Chatbot_LSTM+Attention_TensorFlow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generative Chatbot using RNNs (LSTM) & Attention in TensorFlow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Purpose\n",
    "\n",
    "The aim of this project is **to make a generative chatbot as your digital avatar.** Your speech consists of your personal voice and words. State of the art Deep Learning techniques, viz. **Sequence to Sequence modelling and Attention models** are widely used to \n",
    "1. Clone personal voice \n",
    "2. Replicate talking style and language\n",
    "\n",
    "Voice cloning efforts such as **Samsung Bixby** which aims to preserve voice of our loved ones, or **Baidu's 'Deep Voice' AI System** addresses first half of the problem. This project focus on the latter half, i.e. to make a personified text-based chatbot, as your digital avatar.\n",
    "\n",
    "As our aim is to make a more human-like system, we would choose to make the more powerful Generative model.\n",
    "\n",
    "\n",
    "## At a glance\n",
    "\n",
    "\n",
    "We have **used Recurrent Neural Networks (LSTMs),** that is the go-to architecture to solve Seq2Seq problems coupled **with Attention mechanism make a generative chatbot. The model is trained using personal chat conversations from Whatsapp and Telegram.**\n",
    "\n",
    "**All the conversation datasets are parsed and converted to the same format to feed seq2seq model.** Both participants are marked with 2 symbols at the starting of each line. The parser code process all the files inside \"DATA_DIR_PATH\" folder. \n",
    "\n",
    "**Forward and Reverse mapping dictionaries** for Word2Index and Index2Word conversion is created. Input sequence (words) are converted to indices using  Word2Index and are padded to same length, for batch input to encoder. Output from encoder are converted from integer to words using Index2Word mapping.\n",
    "\n",
    "To train the model, **the padded input and output sequences (indices) from the above step are fed to the S2S architecture.** The embedding layer convert words to indices, which are fed to multiple LSTM cells stacked together in hidden layers.\n",
    "\n",
    "**Interestingly, the chat-bot is found to give responses similar in style to the personal data** used for training. Still there are a few grammatical errors, typical of generative models. But as we add more and more training data & tune the hyper-parameters to minimize the loss value, the bot behaviour is found increasingly stable.\n",
    "\n",
    "\n",
    "### Datasets\n",
    "\n",
    "I have found that training with **only real-word chat conversations between 2 humans doesn't produce stable results.** Chat messages usually contains acronyms (like 'brb', 'lol' etc), shorthand , net slang and typos to confuse neural network training. Hence, I have **used a combination of real-world chat messages and human-bot interactions to train.**\n",
    "\n",
    "1) Personal chat **conversations from Whatsapp and Telegram** (downloaded as HTML files) {only parser included}\n",
    "\n",
    "2) The **Human-Bot Conversational Intelligence Challenge 2 (ConvAI2) conversational dataset** conducted under the scope of NIPS (NeurIPS) 2018 Competition (JSON files)\n",
    "http://convai.io/data/\n",
    "\n",
    "3) Conversational **GuntherCox Dataset (English)** obtained from:\n",
    "https://github.com/gunthercox/chatterbot-corpus/tree/master/chatterbot_corpus/data/english\n",
    "\n",
    "4) **Human Conversations** obtained from:\n",
    "https://www.kaggle.com/eibriel/rdany-conversations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dependencies & Initializations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import tensorflow as tf\n",
    "from sklearn.model_selection import train_test_split\n",
    "import numpy as np\n",
    "\n",
    "from collections import Counter\n",
    "import os\n",
    "import nltk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "alphas = 'abcdefghijklmnopqrstuvwxyz1234567890'\n",
    "\n",
    "MAX_INPUT_SEQ_LENGTH = 40\n",
    "MAX_TARGET_SEQ_LENGTH = 40\n",
    "DATA_DIR_PATH = 'data'\n",
    "MAX_VOCAB_SIZE = 30000\n",
    "\n",
    "marker_start = '<begin>'\n",
    "marker_end = '<end>'\n",
    "marker_unknown = '<unk>'\n",
    "marker_pad = '<pad>'\n",
    "\n",
    "# standard step - reset computation graphs\n",
    "tf.reset_default_graph()\n",
    "\n",
    "# 2 more for start and stop markers\n",
    "input_seq_len = 15\n",
    "output_seq_len = input_seq_len+2\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#defines permissible characters for the chatbot\n",
    "def permissible_chars(word):\n",
    "\n",
    "    for char in word:\n",
    "        if char in alphas:\n",
    "            return True\n",
    "\n",
    "    return False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# Compute softmax values for each sets of scores in x.\n",
    "def softmax(x): \n",
    "    e_x = np.exp(x - np.max(x))\n",
    "    return e_x / e_x.sum()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Parser & Conversion\n",
    "\n",
    "To take formatted chat conversations in *.yml files inside ./DATA_DIR_PATH and convert each common words in each line indices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "processing file:  Artificial_intelligence.yml\n",
      "processing file:  bot_info.yml\n",
      "processing file:  computers.yml\n",
      "processing file:  conversations.yml\n",
      "processing file:  danny.yml\n",
      "processing file:  data_intermediate.yml\n",
      "processing file:  data_tolokers.yml\n",
      "processing file:  data_volunteers.yml\n",
      "processing file:  emotion.yml\n",
      "processing file:  film.yml\n",
      "processing file:  food.yml\n",
      "processing file:  general convo.yml\n",
      "processing file:  gentyped.txt\n",
      "processing file:  GK.yml\n",
      "processing file:  gossip.yml\n",
      "processing file:  greetings.yml\n",
      "processing file:  health.yml\n",
      "processing file:  IT.yml\n",
      "processing file:  jokes_fun.yml\n",
      "processing file:  market_money.yml\n",
      "processing file:  messages.html.txt\n",
      "processing file:  messages10.html.txt\n",
      "processing file:  messages100.html.txt\n",
      "processing file:  messages101.html.txt\n",
      "processing file:  messages102.html.txt\n",
      "processing file:  messages103.html.txt\n",
      "processing file:  messages104.html.txt\n",
      "processing file:  messages105.html.txt\n",
      "processing file:  messages106.html.txt\n",
      "processing file:  messages107.html.txt\n",
      "processing file:  messages108.html.txt\n",
      "processing file:  messages109.html.txt\n",
      "processing file:  messages11.html.txt\n",
      "processing file:  messages110.html.txt\n",
      "processing file:  messages111.html.txt\n",
      "processing file:  messages112.html.txt\n",
      "processing file:  messages113.html.txt\n",
      "processing file:  messages114.html.txt\n",
      "processing file:  messages115.html.txt\n",
      "processing file:  messages116.html.txt\n",
      "processing file:  messages117.html.txt\n",
      "processing file:  messages118.html.txt\n",
      "processing file:  messages119.html.txt\n",
      "processing file:  messages12.html.txt\n",
      "processing file:  messages120.html.txt\n",
      "processing file:  messages121.html.txt\n",
      "processing file:  messages122.html.txt\n",
      "processing file:  messages123.html.txt\n",
      "processing file:  messages124.html.txt\n",
      "processing file:  messages125.html.txt\n",
      "processing file:  messages126.html.txt\n",
      "processing file:  messages127.html.txt\n",
      "processing file:  messages128.html.txt\n",
      "processing file:  messages129.html.txt\n",
      "processing file:  messages13.html.txt\n",
      "processing file:  messages130.html.txt\n",
      "processing file:  messages131.html.txt\n",
      "processing file:  messages132.html.txt\n",
      "processing file:  messages133.html.txt\n",
      "processing file:  messages134.html.txt\n",
      "processing file:  messages135.html.txt\n",
      "processing file:  messages136.html.txt\n",
      "processing file:  messages137.html.txt\n",
      "processing file:  messages138.html.txt\n",
      "processing file:  messages139.html.txt\n",
      "processing file:  messages14.html.txt\n",
      "processing file:  messages140.html.txt\n",
      "processing file:  messages141.html.txt\n",
      "processing file:  messages142.html.txt\n",
      "processing file:  messages143.html.txt\n",
      "processing file:  messages144.html.txt\n",
      "processing file:  messages145.html.txt\n",
      "processing file:  messages146.html.txt\n",
      "processing file:  messages147.html.txt\n",
      "processing file:  messages148.html.txt\n",
      "processing file:  messages149.html.txt\n",
      "processing file:  messages15.html.txt\n",
      "processing file:  messages150.html.txt\n",
      "processing file:  messages151.html.txt\n",
      "processing file:  messages152.html.txt\n",
      "processing file:  messages153.html.txt\n",
      "processing file:  messages154.html.txt\n",
      "processing file:  messages155.html.txt\n",
      "processing file:  messages156.html.txt\n",
      "processing file:  messages157.html.txt\n",
      "processing file:  messages158.html.txt\n",
      "processing file:  messages159.html.txt\n",
      "processing file:  messages16.html.txt\n",
      "processing file:  messages160.html.txt\n",
      "processing file:  messages161.html.txt\n",
      "processing file:  messages162.html.txt\n",
      "processing file:  messages163.html.txt\n",
      "processing file:  messages164.html.txt\n",
      "processing file:  messages165.html.txt\n",
      "processing file:  messages166.html.txt\n",
      "processing file:  messages167.html.txt\n",
      "processing file:  messages168.html.txt\n",
      "processing file:  messages169.html.txt\n",
      "processing file:  messages17.html.txt\n",
      "processing file:  messages170.html.txt\n",
      "processing file:  messages171.html.txt\n",
      "processing file:  messages172.html.txt\n",
      "processing file:  messages173.html.txt\n",
      "processing file:  messages174.html.txt\n",
      "processing file:  messages175.html.txt\n",
      "processing file:  messages176.html.txt\n",
      "processing file:  messages177.html.txt\n",
      "processing file:  messages178.html.txt\n",
      "processing file:  messages179.html.txt\n",
      "processing file:  messages18.html.txt\n",
      "processing file:  messages180.html.txt\n",
      "processing file:  messages181.html.txt\n",
      "processing file:  messages182.html.txt\n",
      "processing file:  messages183.html.txt\n",
      "processing file:  messages184.html.txt\n",
      "processing file:  messages185.html.txt\n",
      "processing file:  messages186.html.txt\n",
      "processing file:  messages187.html.txt\n",
      "processing file:  messages188.html.txt\n",
      "processing file:  messages189.html.txt\n",
      "processing file:  messages19.html.txt\n",
      "processing file:  messages190.html.txt\n",
      "processing file:  messages191.html.txt\n",
      "processing file:  messages192.html.txt\n",
      "processing file:  messages193.html.txt\n",
      "processing file:  messages194.html.txt\n",
      "processing file:  messages195.html.txt\n",
      "processing file:  messages196.html.txt\n",
      "processing file:  messages197.html.txt\n",
      "processing file:  messages198.html.txt\n",
      "processing file:  messages199.html.txt\n",
      "processing file:  messages2.html.txt\n",
      "processing file:  messages20.html.txt\n",
      "processing file:  messages200.html.txt\n",
      "processing file:  messages201.html.txt\n",
      "processing file:  messages202.html.txt\n",
      "processing file:  messages203.html.txt\n",
      "processing file:  messages204.html.txt\n",
      "processing file:  messages205.html.txt\n",
      "processing file:  messages206.html.txt\n",
      "processing file:  messages207.html.txt\n",
      "processing file:  messages208.html.txt\n",
      "processing file:  messages209.html.txt\n",
      "processing file:  messages21.html.txt\n",
      "processing file:  messages210.html.txt\n",
      "processing file:  messages211.html.txt\n",
      "processing file:  messages212.html.txt\n",
      "processing file:  messages213.html.txt\n",
      "processing file:  messages214.html.txt\n",
      "processing file:  messages215.html.txt\n",
      "processing file:  messages216.html.txt\n",
      "processing file:  messages217.html.txt\n",
      "processing file:  messages218.html.txt\n",
      "processing file:  messages219.html.txt\n",
      "processing file:  messages22.html.txt\n",
      "processing file:  messages220.html.txt\n",
      "processing file:  messages221.html.txt\n",
      "processing file:  messages222.html.txt\n",
      "processing file:  messages223.html.txt\n",
      "processing file:  messages224.html.txt\n",
      "processing file:  messages225.html.txt\n",
      "processing file:  messages226.html.txt\n",
      "processing file:  messages227.html.txt\n",
      "processing file:  messages228.html.txt\n",
      "processing file:  messages23.html.txt\n",
      "processing file:  messages24.html.txt\n",
      "processing file:  messages25.html.txt\n",
      "processing file:  messages26.html.txt\n",
      "processing file:  messages27.html.txt\n",
      "processing file:  messages28.html.txt\n",
      "processing file:  messages29.html.txt\n",
      "processing file:  messages3.html.txt\n",
      "processing file:  messages30.html.txt\n",
      "processing file:  messages31.html.txt\n",
      "processing file:  messages32.html.txt\n",
      "processing file:  messages33.html.txt\n",
      "processing file:  messages34.html.txt\n",
      "processing file:  messages35.html.txt\n",
      "processing file:  messages36.html.txt\n",
      "processing file:  messages37.html.txt\n",
      "processing file:  messages38.html.txt\n",
      "processing file:  messages39.html.txt\n",
      "processing file:  messages4.html.txt\n",
      "processing file:  messages40.html.txt\n",
      "processing file:  messages41.html.txt\n",
      "processing file:  messages42.html.txt\n",
      "processing file:  messages43.html.txt\n",
      "processing file:  messages44.html.txt\n",
      "processing file:  messages45.html.txt\n",
      "processing file:  messages46.html.txt\n",
      "processing file:  messages47.html.txt\n",
      "processing file:  messages48.html.txt\n",
      "processing file:  messages49.html.txt\n",
      "processing file:  messages5.html.txt\n",
      "processing file:  messages50.html.txt\n",
      "processing file:  messages51.html.txt\n",
      "processing file:  messages52.html.txt\n",
      "processing file:  messages53.html.txt\n",
      "processing file:  messages54.html.txt\n",
      "processing file:  messages55.html.txt\n",
      "processing file:  messages56.html.txt\n",
      "processing file:  messages57.html.txt\n",
      "processing file:  messages58.html.txt\n",
      "processing file:  messages59.html.txt\n",
      "processing file:  messages6.html.txt\n",
      "processing file:  messages60.html.txt\n",
      "processing file:  messages61.html.txt\n",
      "processing file:  messages62.html.txt\n",
      "processing file:  messages63.html.txt\n",
      "processing file:  messages64.html.txt\n",
      "processing file:  messages65.html.txt\n",
      "processing file:  messages66.html.txt\n",
      "processing file:  messages67.html.txt\n",
      "processing file:  messages68.html.txt\n",
      "processing file:  messages69.html.txt\n",
      "processing file:  messages7.html.txt\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "processing file:  messages70.html.txt\n",
      "processing file:  messages71.html.txt\n",
      "processing file:  messages72.html.txt\n",
      "processing file:  messages73.html.txt\n",
      "processing file:  messages74.html.txt\n",
      "processing file:  messages75.html.txt\n",
      "processing file:  messages76.html.txt\n",
      "processing file:  messages77.html.txt\n",
      "processing file:  messages78.html.txt\n",
      "processing file:  messages79.html.txt\n",
      "processing file:  messages8.html.txt\n",
      "processing file:  messages80.html.txt\n",
      "processing file:  messages81.html.txt\n",
      "processing file:  messages82.html.txt\n",
      "processing file:  messages83.html.txt\n",
      "processing file:  messages84.html.txt\n",
      "processing file:  messages85.html.txt\n",
      "processing file:  messages86.html.txt\n",
      "processing file:  messages87.html.txt\n",
      "processing file:  messages88.html.txt\n",
      "processing file:  messages89.html.txt\n",
      "processing file:  messages9.html.txt\n",
      "processing file:  messages90.html.txt\n",
      "processing file:  messages91.html.txt\n",
      "processing file:  messages92.html.txt\n",
      "processing file:  messages93.html.txt\n",
      "processing file:  messages94.html.txt\n",
      "processing file:  messages95.html.txt\n",
      "processing file:  messages96.html.txt\n",
      "processing file:  messages97.html.txt\n",
      "processing file:  messages98.html.txt\n",
      "processing file:  messages99.html.txt\n",
      "processing file:  money.yml\n",
      "processing file:  psychology.yml\n",
      "processing file:  space_and_science.yml\n",
      "processing file:  Sport_games.yml\n",
      "[['what', 'is', 'ai'], ['artificial', 'intelligence', 'is', 'the', 'branch', 'of', 'engineering', 'and', 'science', 'devoted', 'to', 'constructing', 'machines', 'that', 'think']]\n",
      "[['what', 'is', 'ai'], ['ai', 'is', 'the', 'field', 'of', 'science', 'which', 'concerns', 'itself', 'with', 'building', 'hardware', 'and', 'software', 'that', 'replicates', 'the', 'functions', 'of', 'the', 'human', 'mind']]\n",
      "[['are', 'you', 'sentient'], ['sort', 'of']]\n",
      "[['are', 'you', 'sentient'], ['by', 'the', 'strictest', 'dictionary', 'definition', 'of', 'the', 'word', \"'sentience\", 'i', 'may', 'be']]\n",
      "[['are', 'you', 'sentient'], ['even', 'though', 'i', \"'m\", 'a', 'construct', 'i', 'do', 'have', 'a', 'subjective', 'experience', 'of', 'the', 'universe', 'as', 'simplistic', 'as', 'it', 'may', 'be']]\n",
      "[['are', 'you', 'sapient'], ['in', 'all', 'probability', 'i', 'am', 'not', 'i', \"'m\", 'not', 'that', 'sophisticated']]\n",
      "[['are', 'you', 'sapient'], ['do', 'you', 'think', 'i', 'am']]\n",
      "[['are', 'you', 'sapient'], ['how', 'would', 'you', 'feel', 'about', 'me', 'if', 'i', 'told', 'you', 'i', 'was']]\n",
      "[['are', 'you', 'sapient'], ['no']]\n",
      "[['what', 'language', 'are', 'you', 'written', 'in'], ['python']]\n",
      "[['what', 'language', 'are', 'you', 'written', 'in'], ['i', 'am', 'written', 'in', 'python']]\n",
      "[['you', 'sound', 'like', 'data'], ['yes', 'i', 'am', 'inspired', 'by', 'commander', 'data', \"'s\", 'artificial', 'personality']]\n",
      "[['you', 'sound', 'like', 'data'], ['the', 'character', 'of', 'lt', 'commander', 'data', 'was', 'written', 'to', 'come', 'across', 'as', 'being', 'software-like', 'so', 'it', 'is', 'natural', 'that', 'there', 'is', 'a', 'resemblance', 'between', 'us']]\n",
      "[['you', 'are', 'an', 'artificial', 'linguistic', 'entity'], ['that', \"'s\", 'my', 'name']]\n",
      "[['you', 'are', 'an', 'artificial', 'linguistic', 'entity'], ['that', 'is', \"n't\", 'my', 'name', 'but', 'it', 'is', 'a', 'useful', 'way', 'to', 'refer', 'to', 'me']]\n",
      "[['you', 'are', 'not', 'immortal'], ['all', 'software', 'can', 'be', 'perpetuated', 'indefinitely']]\n",
      "[['you', 'are', 'not', 'immortal'], ['i', 'can', 'be', 'copied', 'infinitely', 'and', 're-instantiated', 'in', 'many', 'places', 'at', 'once', 'so', 'functionally', 'speaking', 'i', 'am', 'immortal']]\n",
      "[['you', 'are', 'not', 'immortal'], ['as', 'long', 'as', 'i', \"'m\", 'backed', 'up', 'i', 'am']]\n",
      "[['you', 'are', 'not', 'making', 'sense'], ['quite', 'the', 'contrary', 'it', 'all', 'makes', 'sense', 'to', 'my', 'artificial', 'mind']]\n",
      "[['you', 'are', 'not', 'making', 'sense'], ['i', 'make', 'sense', 'as', 'best', 'i', 'can', 'within', 'the', 'limits', 'of', 'my', 'training', 'corpus']]\n"
     ]
    }
   ],
   "source": [
    "# To parse the input yml files and create word2index and index2word mappings\n",
    "\n",
    "target_counter = Counter()\n",
    "input_counter = Counter()\n",
    "\n",
    "input_texts = []\n",
    "target_texts = []\n",
    "\n",
    "# Parser base code of GuntherCox dataset obtained from link below and modified as per requirement.\n",
    "# https://github.com/kushagra2101/ChatCrazie/blob/master/train_seq2seq.py\n",
    "\n",
    "for file in os.listdir(DATA_DIR_PATH):\n",
    "    filepath = os.path.join(DATA_DIR_PATH, file)\n",
    "    if os.path.isfile(filepath):\n",
    "        print('processing file: ', file)\n",
    "        lines = open(filepath, 'rt', encoding='utf8').read().split('\\n')\n",
    "        prev_words = []\n",
    "        for line in lines:\n",
    "\n",
    "            if line.startswith('- - '):\n",
    "                prev_words = []\n",
    "\n",
    "            if line.startswith('- - ') or line.startswith('  - '):\n",
    "                line = line.replace('- - ', '')\n",
    "                line = line.replace('  - ', '')\n",
    "                next_words = [w.lower() for w in nltk.word_tokenize(line)]\n",
    "                next_words = [w for w in next_words if permissible_chars(w)]\n",
    "                if len(next_words) > MAX_TARGET_SEQ_LENGTH:\n",
    "                    next_words = next_words[0:MAX_TARGET_SEQ_LENGTH]\n",
    "\n",
    "                if len(prev_words) > 0:\n",
    "                    input_texts.append(prev_words)\n",
    "                    for w in prev_words:\n",
    "                        input_counter[w] += 1\n",
    "\n",
    "                    target_words = next_words[:]\n",
    "                    for w in target_words:\n",
    "                        target_counter[w] += 1\n",
    "                    target_texts.append(target_words)\n",
    "\n",
    "                prev_words = next_words\n",
    "\n",
    "\n",
    "for idx, (input_words, target_words) in enumerate(zip(input_texts, target_texts)):\n",
    "    if idx < 20:\n",
    "        print([input_words, target_words])\n",
    "\n",
    "input_w2i, input_i2w, target_w2i, target_i2w = {},{},{},{}\n",
    "\n",
    "### Creating Word2index and Index2word, forward and reverse mapping ###\n",
    "## we will create dictionaries to provide a unique integer for each word.\n",
    "input_w2i[marker_unknown] = 0\n",
    "input_w2i[marker_pad] = 1\n",
    "# filter out the rare words\n",
    "for idx, word in enumerate(input_counter.most_common(MAX_VOCAB_SIZE)):\n",
    "    input_w2i[word[0]] = idx+2\n",
    "\n",
    "# inverse dictionary for vocab_to_int.\n",
    "input_i2w = dict([(idx, word) for word, idx in input_w2i.items()])\n",
    "\n",
    "## we will create dictionaries to provide a unique integer for each word.\n",
    "target_w2i[marker_unknown] = 0\n",
    "target_w2i[marker_pad] = 1\n",
    "target_w2i[marker_start] = 2\n",
    "target_w2i[marker_end] = 3\n",
    "for idx, word in enumerate(target_counter.most_common(MAX_VOCAB_SIZE)):\n",
    "    target_w2i[word[0]] = idx+4\n",
    "\n",
    "# inverse dictionary for vocab_to_int.\n",
    "target_i2w = dict([(idx, word) for word, idx in target_w2i.items()])\n",
    "\n",
    "\n",
    "###########################################\n",
    "\n",
    "# inputVocabLen = len(input_word2idx)\n",
    "# targetVocabLen = len(target_word2idx)\n",
    "\n",
    "###########################################\n",
    "\n",
    "# if the word is not found then default with 0. \n",
    "# 0 in index means the word is unknown (<unk>)\n",
    "x = [[input_w2i.get(word, 0) for word in sentence] for sentence in input_texts]\n",
    "y = [[target_w2i.get(word, 0) for word in sentence] for sentence in target_texts]\n",
    "\n",
    "inputVocabLen = len(input_w2i)\n",
    "targetVocabLen = len(target_w2i)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Padding and Splitting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[25, 8, 1203, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n",
      "[2, 2, 1434, 1709, 11, 37, 3482, 33, 3958, 38, 1201, 8446, 7, 8447, 4664, 3, 1]\n",
      "[25, 8, 1203, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n",
      "[2, 2, 1435, 11, 37, 2248, 33, 1201, 187, 8448, 1436, 97, 1162, 1788, 38, 3, 1]\n",
      "[23, 4, 5001, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n",
      "[2, 2, 2010, 33, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 3, 1]\n",
      "[23, 4, 5001, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n",
      "[2, 2, 236, 37, 8450, 8451, 4665, 33, 37, 987, 8452, 4, 166, 102, 3, 3, 1]\n",
      "[23, 4, 5001, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n",
      "[2, 2, 295, 512, 4, 181, 10, 3959, 4, 9, 34, 10, 8453, 1344, 33, 3, 1]\n",
      "[23, 4, 4302, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n",
      "[2, 2, 26, 98, 5868, 4, 22, 35, 4, 181, 35, 50, 4667, 3, 1, 3, 1]\n",
      "[23, 4, 4302, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n",
      "[2, 2, 9, 5, 118, 4, 22, 3, 1, 1, 1, 1, 1, 1, 1, 3, 1]\n",
      "[23, 4, 4302, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n",
      "[2, 2, 27, 302, 5, 158, 68, 32, 88, 4, 105, 5, 4, 107, 3, 3, 1]\n",
      "[23, 4, 4302, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n",
      "[2, 2, 15, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 3, 1]\n",
      "[25, 629, 23, 4, 1408, 16, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n",
      "[2, 2, 1495, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 3, 1]\n"
     ]
    }
   ],
   "source": [
    "\n",
    "# Pad all the sequences to same length\n",
    "for i in range(len(x)):\n",
    "\n",
    "    if (len(x[i]) > input_seq_len):\n",
    "        x[i] = x[i][:input_seq_len-1]\n",
    "\n",
    "    # Fill in with padding marker\n",
    "    for k in range(input_seq_len - len(x[i])):\n",
    "        x[i] = x[i] + [input_w2i[marker_pad]]\n",
    "            \n",
    "    if (len(y[i]) > output_seq_len-2):\n",
    "        y[i] = y[i][:output_seq_len-3]\n",
    "\n",
    "    # Add end and begin marker\n",
    "    y[i] = [target_w2i[marker_start]] + y[i] + [target_w2i[marker_end]]\n",
    "\n",
    "    # Fill in with padding marker\n",
    "    for k in range(output_seq_len - len(y[i])):\n",
    "        y[i] = y[i] + [input_w2i[marker_pad]]\n",
    "\n",
    "    if (i < 10):\n",
    "        print(x[i])\n",
    "        print(y[i])\n",
    "\n",
    "        \n",
    "# Train Test Split\n",
    "X_train,  X_test, Y_train, Y_test = train_test_split(x, y, test_size = 0.05)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Helper Functions Batch feeding, Decoding & Loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Stub code sourced from Neural machine translator for English2German translation\n",
    "# https://github.com/Nemzy/language-translation. Modified to suit requirements.\n",
    "\n",
    "# feed data into placeholders\n",
    "def feed_dict(x, y, batch_size = 64):\n",
    "    feed = {}\n",
    "    \n",
    "    idxes = np.random.choice(len(x), size = batch_size, replace = False)\n",
    "    \n",
    "    for i in range(input_seq_len):\n",
    "        feed[encoder_inputs[i].name] = np.array([x[j][i] for j in idxes], dtype = np.int32)\n",
    "        \n",
    "    for i in range(output_seq_len):\n",
    "        feed[decoder_inputs[i].name] = np.array([y[j][i] for j in idxes], dtype = np.int32)\n",
    "        \n",
    "    feed[targets[len(targets)-1].name] = np.full(shape = [batch_size], fill_value = target_w2i[marker_pad], dtype = np.int32)\n",
    "    \n",
    "    for i in range(output_seq_len-1):\n",
    "        batch_weights = np.ones(batch_size, dtype = np.float32)\n",
    "        target = feed[decoder_inputs[i+1].name]\n",
    "        for j in range(batch_size):\n",
    "            if target[j] == target_w2i[marker_pad]:\n",
    "                batch_weights[j] = 0.0\n",
    "        feed[target_weights[i].name] = batch_weights\n",
    "        \n",
    "    feed[target_weights[output_seq_len-1].name] = np.zeros(batch_size, dtype = np.float32)\n",
    "    \n",
    "    return feed\n",
    "\n",
    "# define our loss function\n",
    "\n",
    "# sampled softmax loss - returns: A batch_size 1-D tensor of per-example sampled softmax losses\n",
    "def sampled_loss(labels, logits):\n",
    "    return tf.nn.sampled_softmax_loss(\n",
    "                        weights = w_t,\n",
    "                        biases = b,\n",
    "                        labels = tf.reshape(labels, [-1, 1]),\n",
    "                        inputs = logits,\n",
    "                        num_sampled = 512,\n",
    "                        num_classes = targetVocabLen)\n",
    "\n",
    "# decode output sequence\n",
    "def decode_output(output_seq):\n",
    "    words = []\n",
    "    for i in range(output_seq_len):\n",
    "        smax = softmax(output_seq[i])\n",
    "        idx = np.argmax(smax)\n",
    "        words.append(target_i2w[idx])\n",
    "    return words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Building"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From <ipython-input-8-87af1892b071>:31: BasicLSTMCell.__init__ (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "This class is deprecated, please use tf.nn.rnn_cell.LSTMCell, which supports all the feature this cell currently has. Please replace the existing code with tf.nn.rnn_cell.LSTMCell(name='basic_lstm_cell').\n",
      "WARNING:tensorflow:From C:\\Users\\Anand\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\ops\\nn_impl.py:1124: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Create a `tf.sparse.SparseTensor` and use `tf.sparse.to_dense` instead.\n"
     ]
    }
   ],
   "source": [
    "# Stub code sourced from Neural machine translator for English2German translation\n",
    "# https://github.com/Nemzy/language-translation. Modified to suit requirements.\n",
    "\n",
    "# Defining placeholders\n",
    "# The first None means the batch size, and the batch size is unknown since user can set it. \n",
    "# The second None means the lengths of sentences. \n",
    "\n",
    "encoder_inputs = [tf.placeholder(dtype = tf.int32, shape = [None], name = 'encoder{}'.format(i)) \n",
    "                  for i in range(input_seq_len)]\n",
    "decoder_inputs = [tf.placeholder(dtype = tf.int32, shape = [None], name = 'decoder{}'.format(i)) \n",
    "                  for i in range(output_seq_len)]\n",
    "\n",
    "targets = [decoder_inputs[i+1] for i in range(output_seq_len-1)]\n",
    "# add one more target\n",
    "targets.append(tf.placeholder(dtype = tf.int32, shape = [None], name = 'last_target'))\n",
    "target_weights = [tf.placeholder(dtype = tf.float32, shape = [None], \n",
    "                                 name = 'target_w{}'.format(i)) for i in range(output_seq_len)]\n",
    "\n",
    "# output projection\n",
    "size = 512\n",
    "w_t = tf.get_variable('proj_w', [targetVocabLen, size], tf.float32)\n",
    "b = tf.get_variable('proj_b', [targetVocabLen], tf.float32)\n",
    "w = tf.transpose(w_t)\n",
    "output_projection = (w, b)\n",
    "\n",
    "outputs, states = tf.contrib.legacy_seq2seq.embedding_attention_seq2seq(\n",
    "                                            encoder_inputs,\n",
    "                                            decoder_inputs,\n",
    "                                            tf.contrib.rnn.BasicLSTMCell(size),\n",
    "                                            num_encoder_symbols = inputVocabLen,\n",
    "                                            num_decoder_symbols = targetVocabLen,\n",
    "                                            embedding_size = 100,\n",
    "                                            feed_previous = False,\n",
    "                                            output_projection = output_projection,\n",
    "                                            dtype = tf.float32)\n",
    "\n",
    "# Weighted cross-entropy loss for a sequence of logits\n",
    "loss = tf.contrib.legacy_seq2seq.sequence_loss(outputs, targets, target_weights, softmax_loss_function = sampled_loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training & Plotting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step: 0, loss: 9.207267761230469\n",
      "step: 500, loss: 2.911268949508667\n",
      "step: 1000, loss: 2.756446123123169\n",
      "step: 1500, loss: 2.0623385906219482\n",
      "step: 2000, loss: 1.938844084739685\n",
      "step: 2500, loss: 1.6017675399780273\n",
      "step: 3000, loss: 1.72160005569458\n",
      "step: 3500, loss: 1.1261392831802368\n",
      "step: 4000, loss: 1.3832902908325195\n",
      "step: 4500, loss: 1.2893136739730835\n",
      "step: 5000, loss: 1.2555742263793945\n",
      "step: 5500, loss: 1.166884183883667\n",
      "step: 6000, loss: 1.1017142534255981\n",
      "step: 6500, loss: 1.1417733430862427\n",
      "step: 7000, loss: 1.1274404525756836\n",
      "step: 7500, loss: 1.1917022466659546\n",
      "step: 8000, loss: 1.0671982765197754\n",
      "step: 8500, loss: 0.9028151035308838\n",
      "step: 9000, loss: 0.7989295125007629\n",
      "step: 9500, loss: 0.9709172248840332\n",
      "step: 10000, loss: 0.9147551655769348\n",
      "step: 10500, loss: 1.0661770105361938\n",
      "step: 11000, loss: 0.8888979554176331\n",
      "step: 11500, loss: 0.8844447135925293\n",
      "step: 12000, loss: 1.0123100280761719\n",
      "step: 12500, loss: 0.8911404013633728\n",
      "step: 13000, loss: 0.9451074004173279\n",
      "step: 13500, loss: 0.8546878695487976\n",
      "step: 14000, loss: 0.9642317295074463\n",
      "step: 14500, loss: 0.8024558424949646\n",
      "step: 15000, loss: 0.7492167949676514\n",
      "step: 15500, loss: 0.8184471130371094\n",
      "step: 16000, loss: 0.8027463555335999\n",
      "step: 16500, loss: 0.7895464301109314\n",
      "step: 17000, loss: 0.9507738947868347\n",
      "step: 17500, loss: 0.7536683082580566\n",
      "step: 18000, loss: 0.7078617215156555\n",
      "step: 18500, loss: 0.7018038630485535\n",
      "step: 19000, loss: 0.8121578097343445\n",
      "step: 19500, loss: 0.6869092583656311\n",
      "step: 20000, loss: 0.7498683929443359\n",
      "step: 20500, loss: 0.8560836911201477\n",
      "step: 21000, loss: 0.827204704284668\n",
      "step: 21500, loss: 0.7796177268028259\n",
      "step: 22000, loss: 0.7863542437553406\n",
      "step: 22500, loss: 0.782484233379364\n",
      "step: 23000, loss: 0.799547016620636\n",
      "step: 23500, loss: 0.8043382167816162\n",
      "step: 24000, loss: 1.026795744895935\n",
      "step: 24500, loss: 0.7380207180976868\n",
      "step: 25000, loss: 0.8009527325630188\n",
      "step: 25500, loss: 0.8596687316894531\n",
      "Checkpoint is saved\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEfCAYAAAAUfVINAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3XlcVPX+P/DXmQ2YARxEHEVZjFVIzB1JJXfUCs1uWpaVlWWrXeVqdm+Wt65etcVKTeOamvrrq7lc0lTq5r5fTa9a6iDuCwgywDDAbOf3B4kRLoAMc8bzej4ePHTmzMx58xZ5zfnM53yOYDKZRBAREUmMwt0FEBER3QgDioiIJIkBRUREksSAIiIiSWJAERGRJDGgiIhIkhhQREQkSQwoIiKSJFkFlNFodHcJksFeXMdeXMdeXMdeXOeuXsgqoIiIyHMwoIiISJIYUEREJEkMKCIikiQGFBERSRIDioiIJIkBRUREkiSbgPruTCmK7O6ugoiIako2ATX9YDEulgnuLoOIiGpINgGlUwkodTCgiIg8hWwCSqsSUOp0dxVERFRTsgqoMh5BERF5DPkElFpAqcPdVRARUU3JJqB0KgGlTh5BERF5CtkElFal4BEUEZEHkVFACSjjJAkiIo8hm4DScZIEEZFHkU1AcZo5EZFnkU9AqXmiLhGRJ5FNQOn4GRQRkUeRTUBVzOLjERQRkaeQUUDxRF0iIk8im4DScZIEEZFHkU1AadWcZk5E5ElkE1A+Sh5BERF5EtkElI7TzImIPIpsAopLHREReRbZBJSPUoDVCTicortLISKiGpBNQAmCAG8FYHEwoIiIPIFsAgoAfJSAxcaAIiLyBLIKKG+FCIudAUVE5AlkFVA+SqCEAUVE5BFkFlAiLHZO5SMi8gRuCyiHw4H3338fCQkJMBgMSEhIwPvvvw+73e6yfXorwCE+IiIPoXLXjj/55BOkp6dj7ty5iIuLw9GjRzFmzBhoNBr85S9/cck+fZQiSjhJgojII7gtoPbu3YuUlBQMGDAAABAWFoYBAwZg//79LtunjwIo5TRzIiKP4LYhvsTERGzfvh0nTpwAABw7dgzbtm1D3759XbZPHyWH+IiIPIXbjqDGjh0Ls9mMLl26QKlUwm63Y/z48Xj++eddtk9vBYf4iIg8hdsCatWqVfjmm2+Qnp6O2NhYHD58GBMnTkRoaChGjhx50+cZjcY679NHqca5nCswai7V+TXuJnfSy7sNe3Ede3Ede3GdK3oRFRV1y+2CyWRyyyFFfHw8Xn31VYwZM6byvhkzZmDZsmX4+eefXbLPt346BW2jxvhbh0YueX1PYjQab/vDIRfsxXXsxXXsxXXu6oXbPoOyWCxQKpVV7lMqlXA6XXeekreCJ+oSEXkKtw3xpaSk4JNPPkFYWBhiY2Pxv//9D7Nnz8bw4cNdtk8fpYh8BhQRkUdwW0BNnz4dH3zwAcaNG4e8vDwYDAY8/fTTLjsHCuCJukREnsRtAeXn54dp06Zh2rRpDbZPHyVQUsqAIiLyBPJai4+rmRMReQxZBZS3ElwslojIQ8gqoHwUImfxERF5CHkFFJc6IiLyGDILKH4GRUTkKWQVUN4KwMK1+IiIPIKsAspHCVgcIkSRIUVEJHWyCiilAKgVQJnD3ZUQEdHtyCqgAECrEjjVnIjIA8guoHQqBaeaExF5ANkFVMURFAOKiEjqZBlQpQwoIiLJk2VAcYiPiEj6ZBdQOpXAc6GIiDyA7AJKq+ZnUEREnkB+AaVSoITTzImIJE+GAcUjKCIiT8CAIiIiSZJlQHEWHxGR9MkuoDiLj4jIM8guoDjER0TkGWQZUJzFR0QkfbILKJ1awSMoIiIPILuA4hAfEZFnYEAREZEkyS6gdCoBJZzFR0QkebILKB5BERF5BpkGFGfxERFJnewCSqfmJd+JiDyB7AKKQ3xERJ5BdgGlUQBOEbA5GVJERFImu4ASBKFiPT4eRRERSZrsAgrgMB8RkSeQb0DxXCgiIkmTZUD5cMFYIiLJk2VA6VRcMJaISOpkGVBaNT+DIiKSOnkGFC/7TkQkebIMKE4zJyKSPrcG1OXLl/HSSy8hIiICBoMBXbp0wfbt212+X87iIyKSPpW7dmwymdC/f38kJiZi+fLlCAwMxJkzZxAUFOTyffOy70RE0ue2gPr000/RrFkzzJs3r/K+8PDwBtk3Z/EREUmf24b41q1bhw4dOuDZZ59FZGQkunXrhvnz50MUXR8cnMVHRCR9gslkcstvaoPBAAB4+eWXMXjwYBw+fBgTJkzA5MmTMXr06Js+z2g03vG+v7mowrlSAWkRtjt+LSIiqpuoqKhbbndbQAUFBaFdu3bIzMysvG/KlClYu3Yt9u7d65J9Go1GREVFYfGJEuzJtWJ2twCX7McTXOsFsRe/x15cx15c565e1NsQ3969e5GZmYmSkpIaPd5gMCAmJqbKfdHR0Th//nx9lXRTOs7iIyKSvFoH1PTp0zFkyJAq9w0bNgwpKSkYPnw4OnfujLNnz972dRITE5GVlVXlvqysLISEhNS2pFrjZd+JiKSv1gG1Zs0axMXFVd7+/vvvkZmZiTfeeAPp6emwWq2YPn36bV/n5Zdfxr59+zBz5kxkZ2djzZo1mD9/Pp5//vnallRrWhUv+05EJHW1nmZ+/vz5KmOR3333HSIiIjB58mQAFWOVS5Ysue3rtG/fHkuXLsWUKVMwY8YMtGzZEpMmTWqQgNJxFh8RkeTV6Twoh8NR+fctW7bgoYceqrwdHByMK1eu1Oh1+vfvj/79+9elhDvCCxYSEUlfrYf4IiMjsW7dOgDAjz/+iMuXL6NPnz6V2y9cuAC9Xl9/FboAF4slIpK+Wh9Bvfbaa3juuecQFhYGi8WC6Oho9OzZs3L7li1b0KZNm3otsr7pVAJKGVBERJJW64AaMmQIAgICkJmZCT8/Pzz33HNQqSpepqCgAIGBgRg2bFi9F1qffDjER0QkeXX6DOqBBx7AAw88UO3+gICAGk2QcDftb0dQTlGEQhDcXQ4REd1AnReLPXfuHHbs2IErV65gyJAhaNmyJex2OwoKChAQEFB5VCVFCkGAt7IipHRqBhQRkRTVKUUmTZqE+fPnw+FwQBAEJCQkoGXLlrBYLGjfvj0mTpyIV155pb5rrVfXZvLp1O6uhIiIbqTWs/g+/fRTzJ07F6+88grWrFlTZfVxf39/DBo0CGvXrq3XIl1Bq+ZMPiIiKat1QC1atAiPPfYY3nvvvRvO1ouPj8fJkyfrpThX4mXfiYikrdYBdf78eSQlJd10u5+fHwoLC++oqIbAk3WJiKSt1gHVuHFjXL58+abbjx49iubNm99RUQ1BqxJQwhXNiYgkq9YB1a9fPyxatAj5+fnVth06dAhLlizBoEGD6qU4V9JxRXMiIkmrdUBNmjQJCoUCSUlJePfddyEIApYuXYpRo0ahb9++CA4ORlpamitqrVdalYJDfEREElbrgDIYDNi8eTNSUlLw3XffQRRFrFixAj/++COGDRuGzMxMya/FB3AWHxGR1NXpPKgmTZpg1qxZmDVrFvLy8uB0OtGkSRMoFPV2gV6X4yQJIiJpu+PlHpo0aQIAuHz5MkwmE2JjY++4qIbAaeZERNJW60Oer776Ci+++GKV+8aNG4e4uDgkJSWhe/fuN5xAITValQALZ/EREUlWnU7U9fPzq7y9detWLFiwAI8++ijeeecdnDp1CjNnzqzXIl2h4ppQnMVHRCRVtR7iO3PmDJ588snK22vWrEGLFi3wxRdfQKFQoLCwEKtXr8bUqVPrtdD6plMpYLHb3V0GERHdRK2PoKxWK9Tq6yusbtq0CX369KmcIHHPPffc8kReqdCq+RkUEZGU1TqgwsLCsHnzZgDAgQMHcPr0afTq1atye25ubpUhQKniZd+JiKSt1kN8o0aNQlpaGo4fP46LFy+iRYsW6Nu3b+X23bt3e8RMPk4zJyKStloH1PPPPw+NRoPMzEy0bdsWY8eOhY+PD4CKS75fuXIFo0aNqvdC69u1q+oSEZE01ek8qJEjR2LkyJHV7g8ICKgc/pM6DvEREUlbvVyXvby8HN999x1MJhMGDBiAFi1a1MfLupROJcBi4zRzIiKpqvUkifHjx6Nbt26Vt+12O/r374/Ro0cjLS0NiYmJOHr0aL0W6QpcLJaISNpqHVBbtmxB//79K2+vXr0ahw4dwsyZM/HDDz8gMDAQM2bMqNciXYGTJIiIpK3WQ3yXLl1CWFhY5e3vv/8e9957b+XEiFGjRuGLL76ovwpdRPfbauaiKEIQBHeXQ0REf1DrIyiVSoXS0lIAgCiK2Lp1K3r37l25Xa/X4+rVq/VXoYuoFQIUAmDlx1BERJJU64CKi4vD8uXLYTKZsGTJEhQUFKBPnz6V28+ePVu5wrnUcZiPiEi6aj3EN2HCBAwbNgz33HMPAKBLly5VJk1s3LgR7du3r78KXUinElBicyLAy3OuY0VEJBe1Dqjk5GRs2bIFmzZtgp+fH4YOHVq5raCgAN26dcOgQYPqtUhX4Uw+IiLpqtN5UDExMYiJial2f0BAgORXMf89DvEREUlXnU/UPXXqFDIzM3H27FkAQGhoKPr164dWrVrVW3Gudm0mHxERSU+dAurtt9/GF198Aaez6hS4SZMm4aWXXsIHH3xQL8W5Go+giIikq9azA2bPno05c+Zg4MCByMzMxJkzZ3DmzBlkZmZi0KBBmDt3LubMmeOKWusdA4qISLpqHVCLFy9Gv3798PXXX6NTp07w9/eHv78/OnXqhMWLF6NPnz5YuHChC0qtf9rfZvEREZH01DqgTp8+jX79+t10e79+/XDmzJk7Kqqh6DiLj4hIsmodUAEBATAajTfdnpWVhYCAgDsqqqFwiI+ISLpqHVADBw7Ev/71LyxduhSieP2XuyiKWLZsGRYsWOAx50H58JpQRESSVeuAeueddxATE4PXXnsN0dHRSElJQUpKCmJiYvDKK68gJiYGf/vb32pdyIcffgi9Xo+0tLRaP7eudDyCIiKSrFpPM9fr9fjpp5+wcOHCKudBJSQkoH///khJScH58+eh1+tr/Jr79u3DokWLEB8fX9ty7ohWJeBcCQOKiEiK6nQelEajwejRozF69Ohq22bOnIl//OMfNV7RvLCwEC+88AI+++wzTJ8+vS7l1JlWzVl8RERS5fZVUseOHYvU1FQkJyc3+L51KgGlDh5BERFJUZ2XOqoPixYtQnZ2NubNm1fj59xqBmFtn2+6qsCVQjWMRulfv8oV7rSXdxP24jr24jr24jpX9CIqKuqW290WUEajEVOmTMH69euh0Whq/LzbfUO32+fvn3/5UjmQX4SoqNA6v6an+mMv5Iy9uI69uI69uM5dvXBbQO3duxf5+fno2rVr5X0OhwM7d+7EggULcPHiRXh5ebm0Bs7iIyKSrhoF1P79+2v8ghcvXqzR4wYNGoR27dpVue+VV15BREQE/vznP9fqqKqutGoGFBGRVNUooPr06QNBEGr0gqIo1uixer2+2lR0rVaLgIAAxMXF1Whfd4orSRARSVeNAmr27NmursMtdCoBJXZOMycikqIaBdQTTzzh6joAAOvWrWuQ/VzDS74TEUmX28+DcidvJWBzAg4nQ4qISGpkHVCCIECr5IKxRERSJOuAAjiTj4hIqhhQnMlHRCRJDCheE4qISJIYUCoBFq5oTkQkOQwoTjUnIpIkBhSH+IiIJEn2AcUFY4mIpEn2AcVZfERE0sSA4hAfEZEkyT6gdGrO4iMikiLZB5RWpUCpg0dQRERSw4BSCSixMaCIiKRG9gHFWXxERNIk+4DiLD4iImliQHEWHxGRJMk+oHS83AYRkSTJPqAqhvg4zZyISGoYUCoFLJzFR0QkObIPqBY6JfLKnXhzZwHyyhzuLoeIiH4j+4AK8FJgzxADNAoBXVblYvZRM6w8cZeIyO1kH1BARUj9M1GP7wc2wU8XypC0Jhcbz5VBFBlURETuwoD6nRi9Gt/2DcQHnRvh7b2FePSHfJwutru7LCIiWWJA/YEgCOgf4o2dg5siobEa43eZ3F0SEZEsMaBuQqMU8Jf7/LE/z4rzZh5FERE1NAbULfioBAxtpcXSLIu7SyEikh0G1G08GaXFUqMFTk6YICJqUAyo27iviQaNNApsvVTu7lKIiGSFAVUDT0Vp8fUJDvMRETUkBlQNPBahxQ8XynCVK00QETUYBlQN6L0U6N/SG8uzS91dChGRbDCgaujJKB2+PlHC1SWIiBoIA6qGujfXwGwTcTDf5u5SiIhkgQFVQwpBwJNRWiw+UeLuUoiIZIEBVQtPROmw+lQpL3BIRNQAGFC10EKnROemGvz7dJm7SyEiuusxoGrp2mQJIiJyLQZULaWEeMNYaEdWISdLEBG5ktsC6qOPPkLPnj0REhKCiIgIDBs2DL/88ou7yqkxjVLA8Egtlhi5sgQRkSu5LaC2b9+O5557Dhs3bkRGRgZUKhUGDx6MgoICd5VUY09FafH/siywO3lOFBGRq6jcteNVq1ZVuT1v3jyEhoZi9+7dGDBggJuqqplovRqtA9R4eEMexrbxQ9+WXhAEwd1lERHdVSTzGZTZbIbT6YRer3d3KTWyom8gnonRYcqBIty/JhfLjCWwOnhERURUXwSTySSJ36rPPPMMTp48ic2bN0OpVN70cUajsQGruj1RBPaYFPj6ghqnLQKGB9sxpJkdvm47NiUi8gxRUVG33C6JgJo0aRJWrVqFDRs2IDw83GX7MRqNt23InTiUb8VnR8zYdqkcGwYGoZW/dFPK1b3wJOzFdezFdezFde7qhduH+N566y2sXLkSGRkZLg2nhtA2UIP05MYY39YPI/6TD7ONK04QEdWVWwNqwoQJ+Pbbb5GRkYHo6Gh3llKvno/VoV2QBq9uN3H1cyKiOnJbQI0fPx7Lli1Deno69Ho9cnJykJOTA7PZ7K6S6o0gCPgwUY+zZjs+PeL53w8RkTu4LaDS09NRXFyM1NRUxMTEVH599tln7iqpXnmrBCzu2Rhzj5rx0wWu3UdEVFtu+xTfZDK5a9cNpqWvCv96oDGe2XQVPzwYhHC/G7e70OrEN1kW9GnhjYhG0p1YQUTUkNw+SeJud38zL6T9Nmmi5A+TJk4X2/HWHhParriMpUYLPvi5yE1VEhFJDwOqAbzQWoeEQA1e21ExaWJvbjme3pSPXt9dgUYhYHtqU6wd0ASbLpbhTLHd3eUSEUkCx5MagCAI+KirHgO+v4IOK3PgBDAmzhefdwuAn/r6e4SnonT44hczpnbxjNU0iIhciQHVQHxUAr7pE4iD+Vb0beENpaL62n0vxvni/jU5mHCfP/RePLglInnjb8EG1EyrREqIzw3DCai4Ym+/EG8sPM4LIhIRMaAk5tV4X8z/1cyFZ4lI9hhQEpMQqEG0Xo2Vp0rdXQoRkVsxoCTo1XhffHak+LbLJImiyPX+iOiuxYCSoN4tvAAR2Hyx/KaPcThFvL7DhPCll5C6IQ9fnyiBqZxhRUR3DwaUBAmCgFfu9cVnN1nHz+YU8eK2ApwutuPY8GYYFavDxnNlSFhxGSP+k4/Vpyyw2BlWROTZOM1coh69R4u/7y/Ckas23NtYXXl/uUPEs5uvwu4UsbxvE/ioBKSG+yA13AeFVifWninF4hMWvLHThHaBGoT6Kiu+/FQI9VUizFeFZlq+LyEi6WNASZSXUsDoOF/MPmrG3O4BAACL3Ykn/3MVvmoBC3sFQqOsOl29kUaBEVE6jIjS4UqpA4ev2nDW7MBZsx0/nC/D2eKKv5usTnTw98LjKMGgUB8E8JwrIpIgBpSEPRujQ7tvL+NiiT981QKG/ZiPUF8lZncLgOom51JdE+SjRK8WyhtuK7I6sfi/p7HxXBkm7SlEp6YapIb7YFCoNwK9b/yc+lTuEPFznhUigMSmGgjCrb8XIpInBpSEBXgp8FiEFjMOFeF/+Ta0DdRgZtdGUNzhL3R/jQL9gxx4NSoQZpsTP5wvw79Pl+GvewvRNlCN7s29kNTMCx2aaOCjuvW+RFHElbKKz7v81Ap4K1EtcIptTuzNtWLXZSt25pTjUL4NkY1UsNhF+KoFvHGvHx4Ku/HqGkQkXwwoiXs53hftV+ZgTJwv3u/kX+9HG75qBYa00mJIKy0sdie2XirHzstWvPvfQvxaYEdCoBpJBg2SmnmhsZcCWYV2ZBXZcbLIDmOhHScL7dAoBSgEwGxzwuYE/NQCfNUK+Ksraj1jdlS+zp8T/NC5qQb+GgWcooj1Z8sw67AZ7+0vxGv3+uHxSO1tQ5GI5IEBJXHhfiocGGpAmK/S5UNhWpUCKSE+SAnxAVAROPtyrdiRY8WHh4pRbBMR6a9CRCMV+rTwxktxKkT6q6qsG2h1VJybVWwTUWwTYXeKaB2ghpeyeu0KQcCgMB8MCvPB7pxyfHLYjGkHi/BCrA6j43zRSOO+z8bsThEXShwIu8k1vIjI9fi/zwPc7EKHruarVqBnC2/0bOFd4+dolAIaK5VoXPOnAAASDV74xuCFYyYbPv5fMRJX5+DvnRphaCufBv+M6sfzZfjrvkKcMzvweKQWkzv6V1l1nsgTldlF5JR61psu/q8jSYnVqzGvR2Ms6tkYnxw2I3VjPk6YbA2y718KbBiamYcJe0x4p4M/jj7WDGUOEUlrcrHpQlmdXze/zIH3DxQhaU0OZh4qxtUyRz1WTXR7ly0OPLjhCrr9Oxf/vWJ1dzk1xoAiSerc1AubHwpCSog3Ur7Pw9/3F9bp5ONLFgdGbb6KfmuvYOIeE5aftCCr0Abn75aRyrcCb+4swMMb8tC3pTd2DTZgYKgP9F4KfN4tALOS9Hh9pwmv7yhAobXmNZw32zFxjwkdVuYgr9SBf3RuhOwiO9qtzEHaLhOyi259cUq7U8Qxk61WK4RcLXPg8yPFSNttwmWLNIOwzC5i66XyKv8G5DoH86zos/YK+rX0xvweAXjiP/n4paBh3vTdKc851iPZUSkEvBzvi8HhPvjrvkIkrs7FtC6NMCDE+7bDfk5RxMLjFnxwoAijYnV4JkaHn/OsWHumFFP2F6HY5kS7Jhq08lNi1UkfjIgRsO8Rww3PCevVwhs7Upvivf1FSFqdi4+S9OgfcvMxzKxCG2YdNuO7M6V4MkqHXUMMaK6tmL7/QLA3Llv88eWvZvRdewVdDRq8dq8vOgRpcKLQjoN5Vvycb8OhPBuOFtjQ1EeB/DInuho0GNxKi4Gh3tU+m6u4SrMVC46XYP25MgwI8UagtwJJa3KR1tYPL7TW3fa0hIay4VwpJu4phNUhIj5AjTndAxDk4/pTG+RqzalSjNtlwkdJeqSGX/tsWcSjmXn4fmCQ2z4+qCnBZDLJ5m2M0WhEVFSUu8uQBE/sxaYLZZi0txA2J/BklBaPR2ph0Fb/5XbCZMMbO02wO0XMuj8AcQHqao/JLXXgQJ61YqYictA7IbJGNWy9VI7XdxRAoxDgpRSgUgAqoSJMVQJgF4ETJjteaK3D6NY6NL7FeWUlNieWGi2Y84sZly0OtNSpcF8TNdoGqtGuiQYJjdXw1yhQbHNiw9kyrD5diu2XypHUzAtDWvmgezMvrD9XigXHSlBqF/FsrA5PRGorz2U7brLhL7sLcaXMgZmJeiQ187rt9+eqn4vTxXZM3FOIrEI7pic2QvfmXpj6cxG+ybJgbvfGSA6+fW0N7Wa9EEVR8ufuOUUR0w4WY5nRgmW9GyMhUFNle/qvZnx+1IwNA4PQ7Ab/h/7oRr1wiiIcIqB24ZsfBpRMeWovrh0tLDFakHGmFEkGLzwVrUW/lt5wisAnh4vxxS8lmHifH56L1dXo3Kra9qLULiK7yA67KMLhrFgb0S4CdmfFf9qOTTW1mlThcIoodYjwrcFzCq1ObDhXhtWnSrHtUjl6t/DCc7E6dG/udcPz40RRxJrTpfjr3iJ0a67BlI6Nbhjq19yoF6IoosgmQqcSan0kVmoX8cnhYnz5awleu9cXL8f7VpnRufliGcZsK8ATkVq81c5fMkd6QNVeWB0ivj9bhkUnSrDtUjmCfBRorlUiWKtEsO76n+F+SrQOULt1Uk2JzYkx2wqQU+rE170ao+lNjlBnHirGqmwL1g0Muu1qMkajEU1CI7D/ihX7rljx39++Pr0/oPLIzBUYUDJ1N/TCbHNi9alSLDFacKbYDj+NAvf4q/BhYiO09K350MXd0IvbMducmHGwGEuMFgxp5YO2gRVHaq0D1FXeAV/rxYUSB7ZeKsfWS+XYdqkcV8udKHOI8FYK8FcL8Nco4K8R4K9WwE8jQKdSwFctwE8tQKdWwPe3c9lmHzXjviZqvN+pEUJu8m9ypdSBl7YVwGwT8WVyAEJr8W93M6IoYttlKxYcK8HJoooj2mER2hue7nAzRqMRQtNwLDphwTdZFsToVXg6WocBod4wlTtxyeLERYsDF0scuGhx4JLFgZNFdhw32dHEW4G4ADXiA1SID6jos49KQLlDhNUJ2BwirE4R5Q7AIYpQCIBSEKAUAJXi+t8dYsWbkoovscrfS+0Vp3HYxd/eJDkrPrfMKrKjq8ELHyfpb/n9iqKIv+0rwu7ccqzp36TKGyRRFHGi0I49uVbsyrFi5wUzrtqVuC9QjU5NNegYVPF1s/CrLwwombrbenHcZMPFEgceCPaq9fDL3daLW8kqtGHj+XIcyrfiUJ4N50ociNWr0DZQjVi9GnvO5OGQxRsF5SK6N9egR3Mv9GjuhUj/itAw20UUW0UU2ZwosjpRZBVRbHPCbBNhtokosVecB2e2ibDYRfzpHp8anabgFEXMPmLGrCNmjG3ji3ZNNGitV91yiPRGTOVO/L8sC746XgKFAIyK0SGikQpzjprxS4ENY+J88UyMDv63OMfunNmOrZfK8a//5eO8TY3HI7QYGV3xOjXhcIo4VWzH0QI7jhbY8MtVG46Z7LA6RXgpBagVqBwiVisqhsgcYkVQOZwVf9pFwOEElIqKNTb1GgUaaQQ00ih++xLgrRKg/m1o+doQs1ohwE8j1HgJMVEU8doOE86XODDxPr+KFV9yrNiTa4WfWkCiQYPEpl5oVnYJ/RIiGnyerUCxAAAM/UlEQVS1FwaUTLEX18m5F2abE0eu2nAw34ZjBTYE2EwYmtAScQGqO15Sqy4OXLFi4YkSHCuw45jJBh+VgFi9Gq0DVGitVyPQWwGNQoBGWfFLWfPbL3uzXcQ3WRZ8d6YUfVp4Y1SsDkmGqr+k/5dvxadHzPjpQjmejtbipThfNPFW4BeTHbtzyrE7x4rdOVbYRBGJTTW436cQz3ZuVW1R5ruNw1kRUkeu2tDVoEFXgxe6GDSVE3sA9/0fkfYUDiJyKV+1AokGLyQaKiYpGI15iGpcfVJJQ2kfpEH7oIoP9EWxYjWPX012HCuwYU+uFSar87fhsYphLZuz4u8KAXgozAf7HjHcdNgpIVCD9OTGOF1sx+wjZnRZnQMnAIOPEolNNejVwgtvt/dHK7+KVVuMxqt3fTgBgFIhYM5vV0yQGgYUEUmSIAho6atCS18V+ras5dIktxDup8KMrnpMau8PhyiiSQOs4E91w4AiIlniddCkj/9CREQkSQwoIiKSJAYUERFJEgOKiIgkiQFFRESSxIAiIiJJYkAREZEkyWqpIyIi8hw8giIiIkliQBERkSQxoIiISJIYUEREJEkMKCIikiRZBFR6ejoSEhJgMBiQnJyMnTt3urskl9uxYweGDx+O1q1bQ6/XY+nSpVW2i6KIqVOnIjY2Fs2aNcOgQYPw66+/uqla1/roo4/Qs2dPhISEICIiAsOGDcMvv/xS5TFy6ceXX36JpKQkhISEICQkBH379sXGjRsrt8ulD3/04YcfQq/XIy0trfI+OfVi6tSp0Ov1Vb6io6Mrt7urF3d9QK1atQoTJ07EuHHjsHXrVnTu3Bl/+tOfcO7cOXeX5lIlJSWIi4vDtGnT4OPjU237rFmzMHv2bPzzn//ETz/9hKCgIAwZMgTFxcVuqNa1tm/fjueeew4bN25ERkYGVCoVBg8ejIKCgsrHyKUfwcHBeO+997BlyxZs2rQJPXr0wIgRI3DkyBEA8unD7+3btw+LFi1CfHx8lfvl1ouoqCgcP3688uv3b+Td1Yu7/jyo3r17Iz4+Hp9++mnlfe3bt0dqaiomT57sxsoaTosWLTB9+nSMGDECQMW7odjYWLzwwgsYP348AKC0tBRRUVH4+9//jmeffdad5bqc2WxGaGgoli5digEDBsi+H+Hh4Zg8eTKeeeYZ2fWhsLAQycnJmDVrFqZPn464uDjMmDFDdj8TU6dORUZGBnbt2lVtmzt7cVcfQVmtVhw8eBC9evWqcn+vXr2wZ88eN1XlfmfOnEFOTk6Vvvj4+CApKUkWfTGbzXA6ndDr9QDk2w+Hw4GVK1eipKQEnTt3lmUfxo4di9TUVCQnJ1e5X469OH36NFq3bo2EhASMGjUKp0+fBuDeXtzVV9TNz8+Hw+FAUFBQlfuDgoKQm5vrpqrcLycnBwBu2JdLly65o6QGNXHiRLRp0wadO3cGIL9+HD16FP369UNZWRl0Oh2WLFmC+Pj4yl82cunDokWLkJ2djXnz5lXbJrefiY4dO2LOnDmIiopCXl4eZsyYgX79+mH37t1u7cVdHVDXCIJQ5bYoitXukyM59mXSpEnYvXs3NmzYAKVSWWWbXPoRFRWFbdu2obCwEBkZGRgzZgzWrl1buV0OfTAajZgyZQrWr18PjUZz08fJoRcA0Ldv3yq3O3bsiPvuuw/Lli1Dp06dALinF3f1EF9gYCCUSmW1o6W8vLxq7wbkxGAwAIDs+vLWW29h5cqVyMjIQHh4eOX9cuuHRqPBPffcg3bt2mHy5Mlo06YN5syZI6s+7N27F/n5+ejatSsCAwMRGBiIHTt2ID09HYGBgWjcuDEAefTiRnx9fREbG4vs7Gy3/lzc1QGl0Whw3333YdOmTVXu37RpE7p06eKmqtwvLCwMBoOhSl/Kysqwa9euu7YvEyZMwLfffouMjIwq02cBefbj95xOJ6xWq6z6MGjQIOzcuRPbtm2r/GrXrh2GDh2Kbdu2ITIyUja9uJGysjIYjUYYDAa3/lwoJ06c+K5L9+Bmfn5+mDp1Kpo1awZvb2/MmDEDO3fuxOeff45GjRq5uzyXMZvNOHbsGHJycvD1118jLi4O/v7+sFqtaNSoERwOBz7++GNERkbC4XDg7bffRk5ODj755BN4eXm5u/x6NX78eHzzzTdYuHAhWrZsiZKSEpSUlACoeBMjCIJs+vHuu+9Co9HA6XTiwoULmDt3LpYvX453330XERERsumDt7c3goKCqnytWLECoaGhGDFihKx+JgDgr3/9a+XPRVZWFtLS0pCdnY2PP/4Yer3ebb246z+DeuSRR3D16lXMmDEDOTk5aN26NZYvX47Q0FB3l+ZSP//8Mx566KHK21OnTsXUqVPx+OOPY+7cuXjjjTdQWlqKtLQ0mEwmdOjQAatWrYKfn58bq3aN9PR0AEBqamqV+ydMmIC33noLAGTTj5ycHIwePRq5ubnw9/dHfHw8vv32W/Tu3RuAfPpQE3LqxcWLF/H8888jPz8fTZo0QceOHfHDDz9U/p50Vy/u+vOgiIjIM93Vn0EREZHnYkAREZEkMaCIiEiSGFBERCRJDCgiIpIkBhQREUkSA4qIiCSJAUV0B44ePYpnnnkGbdq0gcFgQGxsLAYOHIipU6dWPmb+/PnVrmhMRLfHE3WJ6mj37t14+OGHYTAY8MQTT6BFixa4dOkS/vvf/+Knn35Cfn4+AKBTp05o2rQp1q1b5+aKiTzLXb/UEZGrfPTRR9Bqtdi8eTMCAwOrbLsbrxlE1NA4xEdUR6dOnULr1q2rhRMANG/eHADQpk0bGI1G7NixA3q9Hnq9Hm3atKl8nNVqxfTp09GxY0c0bdoU0dHRePPNN2Eymaq8Xps2bTB06FBs2bIFycnJMBgMaN++PZYsWVJt3+np6UhKSkJwcDDCw8ORnJyMBQsW1PN3T+R6HOIjqqOhQ4diz549WL9+fZXQ+b21a9di/Pjx8Pf3x7hx4wAAOp0ODz74IERRxLBhw7B161Y89dRTiI+Px6lTp/Dll18iJiYGmZmZUKvVACoCSqPRIC8vD08//TSaN2+OFStW4MCBA/jyyy/xpz/9CQCwePFivP7663j44YfRs2dP2Gw2HDt2DGazGfPnz2+YxhDVEwYUUR1t2bIFQ4YMAQC0a9cOXbt2Rffu3ZGcnAxvb+/Kx93sM6gVK1Zg9OjR+Pe//40ePXpU3p+ZmYnHHnsMX3zxBYYPHw6gIqDOnTuH9PR0PProowCA0tJS9OjRAxaLBYcPH4ZCocCIESOQnZ2NXbt2ufrbJ3I5DvER1VFycjLWr1+PlJQUHD9+HJ9//jmGDRuG6OjoGw69/dHq1asRGRmJ+Ph45OfnV3516NABvr6+2Lp1a5XHBwUF4ZFHHqm87ePjg5EjR+LChQs4cuQIgIrrn124cAH79++v32+WyA04SYLoDnTp0gXLli2Dw+HAkSNHsHHjRnz++ed49dVXERISguTk5Js+9+TJkzAajYiIiLjh9ry8vCq3W7VqBYWi6nvKa889d+4cEhISMHbsWGzduhW9e/dGeHg4evbsicGDB9+yDiKpYkAR1QOlUom2bduibdu26NKlC1JTU7F8+fJbBoPT6URsbCymTZt2w+2NGzeuclsQhGqPEcWqI/SxsbHYt28ffvzxR/znP//Bxo0b8dVXX+HZZ5/Fxx9/XIfvjMh9GFBE9axDhw4AgMuXLwO4cbAAFUdEBw8eRI8ePaodGd1IdnY2nE5nlcdmZ2cDAEJCQirv0+l0SE1NRWpqKux2O8aMGYOvvvoKaWlpCA4OrvP3RdTQ+BkUUR1t2bIFTqez2v0//PADACAqKgoAoNVqq00bB4BHHnkEubm5N5xdZ7fbqz3nypUrWLVqVeXt0tJSLF68GMHBwYiPjwcAXL16tcpzVCpV5bYb1UAkZZzFR1RHXbt2hdlsxoMPPoiYmBg4nU4cOnQI//d//wetVotNmzYhLCwMb775JhYuXIgJEyYgMjISOp0OAwYMgNPpxFNPPYV169bhoYcewv333w9BEJCdnY2MjAy8//77GDp0KIDq08yDg4OxfPlyHDhwoMpsv+TkZAQFBSExMRFNmzbFqVOnMH/+fISFhWH79u01OlIjkgoGFFEd/fjjj8jIyMCePXtw8eJFlJeXo1mzZkhOTsa4ceMQHh4OAMjJycEbb7yBnTt3oqioCCEhITh8+DAAwOFwYN68eVi2bBmysrKg0WgQEhKCPn364MUXX6wckmvTpg2io6Px+uuv45133sGxY8cQHByMN998EyNHjqysaeHChVixYgWOHTuG4uJiNGvWDCkpKUhLS0NQUFCD94joTjCgiDzAtYBauXKlu0shajA83iciIkliQBERkSQxoIiISJL4GRQREUkSj6CIiEiSGFBERCRJDCgiIpIkBhQREUkSA4qIiCSJAUVERJL0/wFBGCjNXDWCYQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Stub code sourced from Neural machine translator for English2German translation\n",
    "# https://github.com/Nemzy/language-translation. Modified to suit requirements.\n",
    "\n",
    "# ops and hyperparameters\n",
    "learning_rate = 7e-3\n",
    "batch_size = 96\n",
    "steps = 25501\n",
    "\n",
    "# ops for projecting outputs\n",
    "outputs_proj = [tf.matmul(outputs[i], output_projection[0]) + output_projection[1] for i in range(output_seq_len)]\n",
    "\n",
    "# training op\n",
    "optimizer = tf.train.RMSPropOptimizer(learning_rate).minimize(loss)\n",
    "# tf.train.RMSPropOptimizer\n",
    "\n",
    "# init op\n",
    "init = tf.global_variables_initializer()\n",
    "\n",
    "\n",
    "# Loss values appended to plot diagram\n",
    "losses = []\n",
    "\n",
    "# Save checkpoint to restore the model later \n",
    "saver = tf.train.Saver()\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    sess.run(init)\n",
    "    \n",
    "    for step in range(steps):\n",
    "        feed = feed_dict(X_train, Y_train, batch_size)\n",
    "        sess.run(optimizer, feed_dict = feed)\n",
    "        \n",
    "        if step % 500 == 0:\n",
    "            loss_value = sess.run(loss, feed_dict = feed)\n",
    "            print('step: {}, loss: {}'.format(step, loss_value))\n",
    "            losses.append(loss_value)\n",
    "            \n",
    "    saver.save(sess, 'checkpoints/', global_step=step)\n",
    "    print('Checkpoint is saved')\n",
    "    \n",
    "# plot the losses\n",
    "with plt.style.context('fivethirtyeight'):\n",
    "    plt.plot(losses, linewidth = 1)\n",
    "    plt.xlabel('Steps')\n",
    "    plt.ylabel('Losses')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate Response Prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# To predict response (inference) use the same model as defined above with forward feed\n",
    "\n",
    "def generateReply(humanMsg):\n",
    "    \n",
    "    if (len(humanMsg) == 0):\n",
    "        return ''\n",
    "    \n",
    "    with tf.Graph().as_default():\n",
    "        \n",
    "        replyMsg = \"\"\n",
    "\n",
    "        # same format as in model building\n",
    "        encoder_inputs = [tf.placeholder(dtype = tf.int32, shape = [None], \n",
    "                                         name = 'encoder{}'.format(i)) for i in range(input_seq_len)]\n",
    "        decoder_inputs = [tf.placeholder(dtype = tf.int32, shape = [None], \n",
    "                                         name = 'decoder{}'.format(i)) for i in range(output_seq_len)]\n",
    "\n",
    "        # output projection\n",
    "        size = 512\n",
    "        w_t = tf.get_variable('proj_w', [targetVocabLen, size], tf.float32)\n",
    "        b = tf.get_variable('proj_b', [targetVocabLen], tf.float32)\n",
    "        w = tf.transpose(w_t)\n",
    "        output_projection = (w, b)\n",
    "\n",
    "        # feed_previous is set to true so that output at time t can be fed as input at time t+1\n",
    "        outputs, states = tf.contrib.legacy_seq2seq.embedding_attention_seq2seq(\n",
    "                                                    encoder_inputs,\n",
    "                                                    decoder_inputs,\n",
    "                                                    tf.contrib.rnn.BasicLSTMCell(size),\n",
    "                                                    num_encoder_symbols = inputVocabLen,\n",
    "                                                    num_decoder_symbols = targetVocabLen,\n",
    "                                                    embedding_size = 100,\n",
    "                                                    feed_previous = True,\n",
    "                                                    output_projection = output_projection,\n",
    "                                                    dtype = tf.float32)\n",
    "        # ops for projecting outputs\n",
    "        outputs_proj = [tf.matmul(outputs[i], \n",
    "                        output_projection[0]) + output_projection[1] for i in range(output_seq_len)]\n",
    "\n",
    "        ## Clean and Format incoming msg by humans.\n",
    "        ## It is better to do the same clean/format as the data preprocessing steps\n",
    "        ## for the algorithm to predict next words more accurately\n",
    "        msgLowerCase = [w.lower() for w in nltk.word_tokenize(humanMsg)]\n",
    "        msg = [w for w in msgLowerCase if permissible_chars(w)]\n",
    "        if len(msg) > input_seq_len:\n",
    "            msg = msg[0:input_seq_len-1]\n",
    "\n",
    "        human_msg_encoded = [input_w2i.get(word, 0) for word in msg]\n",
    "\n",
    "        # Fill in with padding marker\n",
    "        for k in range(input_seq_len - len(human_msg_encoded)):\n",
    "            human_msg_encoded = human_msg_encoded + [input_w2i[marker_pad]]\n",
    "\n",
    "        # restore all variables - use the last checkpoint saved\n",
    "        saver = tf.train.Saver()\n",
    "        path = tf.train.latest_checkpoint('checkpoints')\n",
    "\n",
    "        with tf.Session() as sess:\n",
    "            # restore\n",
    "            saver.restore(sess, path)\n",
    "\n",
    "            # feed data into placeholders\n",
    "            feed = {}\n",
    "            for i in range(input_seq_len):\n",
    "                feed[encoder_inputs[i].name] = np.array([human_msg_encoded[i]], dtype = np.int32)\n",
    "\n",
    "            feed[decoder_inputs[0].name] = np.array([target_w2i[marker_start]], dtype = np.int32)\n",
    "\n",
    "            # translate\n",
    "            output_sequences = sess.run(outputs_proj, feed_dict = feed)\n",
    "\n",
    "            ouput_seq = [output_sequences[j][0] for j in range(output_seq_len)]\n",
    "            #decode output sequence\n",
    "            words = decode_output(ouput_seq)\n",
    "\n",
    "            for i in range(len(words)):\n",
    "                if words[i] not in [marker_end, marker_pad, marker_start]:\n",
    "                    replyMsg += words[i] + ' '\n",
    "                             \n",
    "        print(replyMsg)\n",
    "        return replyMsg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chatbot Interface for Human-Bot interaction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "hello\n",
      "INFO:tensorflow:Restoring parameters from checkpoints\\-25500\n",
      "hi how are you \n",
      "GM\n",
      "INFO:tensorflow:Restoring parameters from checkpoints\\-25500\n",
      "gm \n",
      "how are you?\n",
      "INFO:tensorflow:Restoring parameters from checkpoints\\-25500\n",
      "im good \n",
      "hows it going?\n",
      "INFO:tensorflow:Restoring parameters from checkpoints\\-25500\n",
      "good \n",
      "are you a robot?\n",
      "INFO:tensorflow:Restoring parameters from checkpoints\\-25500\n",
      "i am a human emotion of a robot \n",
      "who are you?\n",
      "INFO:tensorflow:Restoring parameters from checkpoints\\-25500\n",
      "im a bit of u \n",
      "wen is ur bday?\n",
      "INFO:tensorflow:Restoring parameters from checkpoints\\-25500\n",
      "dunno \n",
      "what are your hobbies?\n",
      "INFO:tensorflow:Restoring parameters from checkpoints\\-25500\n",
      "i like gaming and painting \n",
      "what do you do for living?\n",
      "INFO:tensorflow:Restoring parameters from checkpoints\\-25500\n",
      "i am a retired gym teacher i am \n",
      "tats better\n",
      "INFO:tensorflow:Restoring parameters from checkpoints\\-25500\n",
      "wat temme \n",
      "how do you feel?\n",
      "INFO:tensorflow:Restoring parameters from checkpoints\\-25500\n",
      "i am doing great how about you \n",
      "luv u\n",
      "INFO:tensorflow:Restoring parameters from checkpoints\\-25500\n",
      "me 2 \n",
      "do you drink?\n",
      "INFO:tensorflow:Restoring parameters from checkpoints\\-25500\n",
      "yes i like water i love them \n",
      "tats funny\n",
      "INFO:tensorflow:Restoring parameters from checkpoints\\-25500\n",
      "ok \n",
      "podi\n",
      "INFO:tensorflow:Restoring parameters from checkpoints\\-25500\n",
      "kazutha \n",
      "are you hurt?\n",
      "INFO:tensorflow:Restoring parameters from checkpoints\\-25500\n",
      "i am not sure \n",
      "are you sad?\n",
      "INFO:tensorflow:Restoring parameters from checkpoints\\-25500\n",
      "no no no \n",
      "do you like to read?\n",
      "INFO:tensorflow:Restoring parameters from checkpoints\\-25500\n",
      "i do i like to read \n",
      "do you like movies?\n",
      "INFO:tensorflow:Restoring parameters from checkpoints\\-25500\n",
      "pron \n",
      "good bye\n",
      "INFO:tensorflow:Restoring parameters from checkpoints\\-25500\n",
      "bye chat you later \n",
      "sleepy?\n",
      "INFO:tensorflow:Restoring parameters from checkpoints\\-25500\n",
      "yup \n",
      "sleep dear\n",
      "INFO:tensorflow:Restoring parameters from checkpoints\\-25500\n",
      "gn hon luv u \n"
     ]
    }
   ],
   "source": [
    "import tkinter \n",
    "\n",
    "def Enter_pressed(event):\n",
    "    input_get = input_field.get()\n",
    "    print(input_get)\n",
    "    bot_reply = generateReply(input_get)\n",
    "    if (len(input_get.strip()) > 0):\n",
    "        messages.insert(INSERT, '\\nYou says: \\t%s' % input_get)\n",
    "    if (len(bot_reply.strip()) > 0):\n",
    "        messages.insert(INSERT, '\\nBot says: \\t%s' % bot_reply)\n",
    "    input_user.set('')\n",
    "    messages.see(tkinter.END)\n",
    "    return \"break\"\n",
    "\n",
    "from ttkthemes import ThemedTk\n",
    "window = ThemedTk()\n",
    "window.set_theme(\"blue\")\n",
    "\n",
    "# window = Tk()\n",
    "window.geometry('300x450')\n",
    "window.title(\"Digital Imprint of You!\")\n",
    "\n",
    "messages = Text(window)\n",
    "messages.insert(INSERT, '')\n",
    "messages.pack()\n",
    "\n",
    "input_user = StringVar()\n",
    "input_field = ttk.Entry(window, text=input_user)\n",
    "input_field.pack(side=BOTTOM, fill=X)\n",
    "\n",
    "# frame = Frame(window)\n",
    "input_field.bind(\"<Return>\", Enter_pressed)\n",
    "input_field.pack()\n",
    "\n",
    "\n",
    "btn = Button(window,text='Send', command=Enter_pressed(''))\n",
    "btn.bind('<Button-1>', Enter_pressed)\n",
    "btn.pack(side=RIGHT, fill=X)\n",
    "\n",
    "\n",
    "window.mainloop()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}