summarize_reviews.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Summarizing Text with Amazon Reviews"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The objective of this project is to build a model that can create relevant summaries for reviews written about fine foods sold on Amazon. This dataset contains above 500,000 reviews, and is hosted on [Kaggle.](https://www.kaggle.com/snap/amazon-fine-food-reviews)\n",
    "\n",
    "To build our model we will use a two-layered bidirectional RNN with LSTMs on the input data and two layers, each with an LSTM using bahdanau attention on the target data. [Jaemin Cho's tutorial](https://github.com/j-min/tf_tutorial_plus/tree/master/RNN_seq2seq/contrib_seq2seq) for seq2seq was really helpful to get the code in working order because this is my first project with TensorFlow 1.1; some of the functions are very different from 1.0. The architecture for this model is similar to Xin Pan's and Peter Liu's, here's their [GitHub page.](https://github.com/tensorflow/models/tree/master/textsum)\n",
    "\n",
    "The sections of this project are:\n",
    "- Inspecting the Data\n",
    "- Preparing the Data\n",
    "- Building the Model\n",
    "- Training the Model\n",
    "- Making Our Own Summaries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TensorFlow Version: 1.1.0\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import re\n",
    "from nltk.corpus import stopwords\n",
    "import time\n",
    "from tensorflow.python.layers.core import Dense\n",
    "from tensorflow.python.ops.rnn_cell_impl import _zero_state_tensors\n",
    "print('TensorFlow Version: {}'.format(tf.__version__))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Insepcting the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "reviews = pd.read_csv(\"Reviews.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(568454, 10)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>ProductId</th>\n",
       "      <th>UserId</th>\n",
       "      <th>ProfileName</th>\n",
       "      <th>HelpfulnessNumerator</th>\n",
       "      <th>HelpfulnessDenominator</th>\n",
       "      <th>Score</th>\n",
       "      <th>Time</th>\n",
       "      <th>Summary</th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>B001E4KFG0</td>\n",
       "      <td>A3SGXH7AUHU8GW</td>\n",
       "      <td>delmartian</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>1303862400</td>\n",
       "      <td>Good Quality Dog Food</td>\n",
       "      <td>I have bought several of the Vitality canned d...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>B00813GRG4</td>\n",
       "      <td>A1D87F6ZCVE5NK</td>\n",
       "      <td>dll pa</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1346976000</td>\n",
       "      <td>Not as Advertised</td>\n",
       "      <td>Product arrived labeled as Jumbo Salted Peanut...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>B000LQOCH0</td>\n",
       "      <td>ABXLMWJIXXAIN</td>\n",
       "      <td>Natalia Corres \"Natalia Corres\"</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>1219017600</td>\n",
       "      <td>\"Delight\" says it all</td>\n",
       "      <td>This is a confection that has been around a fe...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>B000UA0QIQ</td>\n",
       "      <td>A395BORC6FGVXV</td>\n",
       "      <td>Karl</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>1307923200</td>\n",
       "      <td>Cough Medicine</td>\n",
       "      <td>If you are looking for the secret ingredient i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>B006K2ZZ7K</td>\n",
       "      <td>A1UQRSCLF8GW1T</td>\n",
       "      <td>Michael D. Bigham \"M. Wassir\"</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>1350777600</td>\n",
       "      <td>Great taffy</td>\n",
       "      <td>Great taffy at a great price.  There was a wid...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id   ProductId          UserId                      ProfileName  \\\n",
       "0   1  B001E4KFG0  A3SGXH7AUHU8GW                       delmartian   \n",
       "1   2  B00813GRG4  A1D87F6ZCVE5NK                           dll pa   \n",
       "2   3  B000LQOCH0   ABXLMWJIXXAIN  Natalia Corres \"Natalia Corres\"   \n",
       "3   4  B000UA0QIQ  A395BORC6FGVXV                             Karl   \n",
       "4   5  B006K2ZZ7K  A1UQRSCLF8GW1T    Michael D. Bigham \"M. Wassir\"   \n",
       "\n",
       "   HelpfulnessNumerator  HelpfulnessDenominator  Score        Time  \\\n",
       "0                     1                       1      5  1303862400   \n",
       "1                     0                       0      1  1346976000   \n",
       "2                     1                       1      4  1219017600   \n",
       "3                     3                       3      2  1307923200   \n",
       "4                     0                       0      5  1350777600   \n",
       "\n",
       "                 Summary                                               Text  \n",
       "0  Good Quality Dog Food  I have bought several of the Vitality canned d...  \n",
       "1      Not as Advertised  Product arrived labeled as Jumbo Salted Peanut...  \n",
       "2  \"Delight\" says it all  This is a confection that has been around a fe...  \n",
       "3         Cough Medicine  If you are looking for the secret ingredient i...  \n",
       "4            Great taffy  Great taffy at a great price.  There was a wid...  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Id                         0\n",
       "ProductId                  0\n",
       "UserId                     0\n",
       "ProfileName               16\n",
       "HelpfulnessNumerator       0\n",
       "HelpfulnessDenominator     0\n",
       "Score                      0\n",
       "Time                       0\n",
       "Summary                   26\n",
       "Text                       0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check for any nulls values\n",
    "reviews.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Remove null values and unneeded features\n",
    "reviews = reviews.dropna()\n",
    "reviews = reviews.drop(['Id','ProductId','UserId','ProfileName','HelpfulnessNumerator','HelpfulnessDenominator',\n",
    "                        'Score','Time'], 1)\n",
    "reviews = reviews.reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Summary</th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Good Quality Dog Food</td>\n",
       "      <td>I have bought several of the Vitality canned d...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Not as Advertised</td>\n",
       "      <td>Product arrived labeled as Jumbo Salted Peanut...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\"Delight\" says it all</td>\n",
       "      <td>This is a confection that has been around a fe...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Cough Medicine</td>\n",
       "      <td>If you are looking for the secret ingredient i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Great taffy</td>\n",
       "      <td>Great taffy at a great price.  There was a wid...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 Summary                                               Text\n",
       "0  Good Quality Dog Food  I have bought several of the Vitality canned d...\n",
       "1      Not as Advertised  Product arrived labeled as Jumbo Salted Peanut...\n",
       "2  \"Delight\" says it all  This is a confection that has been around a fe...\n",
       "3         Cough Medicine  If you are looking for the secret ingredient i...\n",
       "4            Great taffy  Great taffy at a great price.  There was a wid..."
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Review # 1\n",
      "Good Quality Dog Food\n",
      "I have bought several of the Vitality canned dog food products and have found them all to be of good quality. The product looks more like a stew than a processed meat and it smells better. My Labrador is finicky and she appreciates this product better than  most.\n",
      "\n",
      "Review # 2\n",
      "Not as Advertised\n",
      "Product arrived labeled as Jumbo Salted Peanuts...the peanuts were actually small sized unsalted. Not sure if this was an error or if the vendor intended to represent the product as \"Jumbo\".\n",
      "\n",
      "Review # 3\n",
      "\"Delight\" says it all\n",
      "This is a confection that has been around a few centuries.  It is a light, pillowy citrus gelatin with nuts - in this case Filberts. And it is cut into tiny squares and then liberally coated with powdered sugar.  And it is a tiny mouthful of heaven.  Not too chewy, and very flavorful.  I highly recommend this yummy treat.  If you are familiar with the story of C.S. Lewis' \"The Lion, The Witch, and The Wardrobe\" - this is the treat that seduces Edmund into selling out his Brother and Sisters to the Witch.\n",
      "\n",
      "Review # 4\n",
      "Cough Medicine\n",
      "If you are looking for the secret ingredient in Robitussin I believe I have found it.  I got this in addition to the Root Beer Extract I ordered (which was good) and made some cherry soda.  The flavor is very medicinal.\n",
      "\n",
      "Review # 5\n",
      "Great taffy\n",
      "Great taffy at a great price.  There was a wide assortment of yummy taffy.  Delivery was very quick.  If your a taffy lover, this is a deal.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Inspecting some of the reviews\n",
    "for i in range(5):\n",
    "    print(\"Review #\",i+1)\n",
    "    print(reviews.Summary[i])\n",
    "    print(reviews.Text[i])\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# A list of contractions from http://stackoverflow.com/questions/19790188/expanding-english-language-contractions-in-python\n",
    "contractions = { \n",
    "\"ain't\": \"am not\",\n",
    "\"aren't\": \"are not\",\n",
    "\"can't\": \"cannot\",\n",
    "\"can't've\": \"cannot have\",\n",
    "\"'cause\": \"because\",\n",
    "\"could've\": \"could have\",\n",
    "\"couldn't\": \"could not\",\n",
    "\"couldn't've\": \"could not have\",\n",
    "\"didn't\": \"did not\",\n",
    "\"doesn't\": \"does not\",\n",
    "\"don't\": \"do not\",\n",
    "\"hadn't\": \"had not\",\n",
    "\"hadn't've\": \"had not have\",\n",
    "\"hasn't\": \"has not\",\n",
    "\"haven't\": \"have not\",\n",
    "\"he'd\": \"he would\",\n",
    "\"he'd've\": \"he would have\",\n",
    "\"he'll\": \"he will\",\n",
    "\"he's\": \"he is\",\n",
    "\"how'd\": \"how did\",\n",
    "\"how'll\": \"how will\",\n",
    "\"how's\": \"how is\",\n",
    "\"i'd\": \"i would\",\n",
    "\"i'll\": \"i will\",\n",
    "\"i'm\": \"i am\",\n",
    "\"i've\": \"i have\",\n",
    "\"isn't\": \"is not\",\n",
    "\"it'd\": \"it would\",\n",
    "\"it'll\": \"it will\",\n",
    "\"it's\": \"it is\",\n",
    "\"let's\": \"let us\",\n",
    "\"ma'am\": \"madam\",\n",
    "\"mayn't\": \"may not\",\n",
    "\"might've\": \"might have\",\n",
    "\"mightn't\": \"might not\",\n",
    "\"must've\": \"must have\",\n",
    "\"mustn't\": \"must not\",\n",
    "\"needn't\": \"need not\",\n",
    "\"oughtn't\": \"ought not\",\n",
    "\"shan't\": \"shall not\",\n",
    "\"sha'n't\": \"shall not\",\n",
    "\"she'd\": \"she would\",\n",
    "\"she'll\": \"she will\",\n",
    "\"she's\": \"she is\",\n",
    "\"should've\": \"should have\",\n",
    "\"shouldn't\": \"should not\",\n",
    "\"that'd\": \"that would\",\n",
    "\"that's\": \"that is\",\n",
    "\"there'd\": \"there had\",\n",
    "\"there's\": \"there is\",\n",
    "\"they'd\": \"they would\",\n",
    "\"they'll\": \"they will\",\n",
    "\"they're\": \"they are\",\n",
    "\"they've\": \"they have\",\n",
    "\"wasn't\": \"was not\",\n",
    "\"we'd\": \"we would\",\n",
    "\"we'll\": \"we will\",\n",
    "\"we're\": \"we are\",\n",
    "\"we've\": \"we have\",\n",
    "\"weren't\": \"were not\",\n",
    "\"what'll\": \"what will\",\n",
    "\"what're\": \"what are\",\n",
    "\"what's\": \"what is\",\n",
    "\"what've\": \"what have\",\n",
    "\"where'd\": \"where did\",\n",
    "\"where's\": \"where is\",\n",
    "\"who'll\": \"who will\",\n",
    "\"who's\": \"who is\",\n",
    "\"won't\": \"will not\",\n",
    "\"wouldn't\": \"would not\",\n",
    "\"you'd\": \"you would\",\n",
    "\"you'll\": \"you will\",\n",
    "\"you're\": \"you are\"\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 203,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def clean_text(text, remove_stopwords = True):\n",
    "    '''Remove unwanted characters, stopwords, and format the text to create fewer nulls word embeddings'''\n",
    "    \n",
    "    # Convert words to lower case\n",
    "    text = text.lower()\n",
    "    \n",
    "    # Replace contractions with their longer forms \n",
    "    if True:\n",
    "        text = text.split()\n",
    "        new_text = []\n",
    "        for word in text:\n",
    "            if word in contractions:\n",
    "                new_text.append(contractions[word])\n",
    "            else:\n",
    "                new_text.append(word)\n",
    "        text = \" \".join(new_text)\n",
    "    \n",
    "    # Format words and remove unwanted characters\n",
    "    text = re.sub(r'https?:\\/\\/.*[\\r\\n]*', '', text, flags=re.MULTILINE)\n",
    "    text = re.sub(r'\\<a href', ' ', text)\n",
    "    text = re.sub(r'&amp;', '', text) \n",
    "    text = re.sub(r'[_\"\\-;%()|+&=*%.,!?:#$@\\[\\]/]', ' ', text)\n",
    "    text = re.sub(r'<br />', ' ', text)\n",
    "    text = re.sub(r'\\'', ' ', text)\n",
    "    \n",
    "    # Optionally, remove stop words\n",
    "    if remove_stopwords:\n",
    "        text = text.split()\n",
    "        stops = set(stopwords.words(\"english\"))\n",
    "        text = [w for w in text if not w in stops]\n",
    "        text = \" \".join(text)\n",
    "\n",
    "    return text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will remove the stopwords from the texts because they do not provide much use for training our model. However, we will keep them for our summaries so that they sound more like natural phrases. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 204,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Summaries are complete.\n",
      "Texts are complete.\n"
     ]
    }
   ],
   "source": [
    "# Clean the summaries and texts\n",
    "clean_summaries = []\n",
    "for summary in reviews.Summary:\n",
    "    clean_summaries.append(clean_text(summary, remove_stopwords=False))\n",
    "print(\"Summaries are complete.\")\n",
    "\n",
    "clean_texts = []\n",
    "for text in reviews.Text:\n",
    "    clean_texts.append(clean_text(text))\n",
    "print(\"Texts are complete.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 206,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Clean Review # 1\n",
      "good quality dog food\n",
      "bought several vitality canned dog food products found good quality product looks like stew processed meat smells better labrador finicky appreciates product better\n",
      "\n",
      "Clean Review # 2\n",
      "not as advertised\n",
      "product arrived labeled jumbo salted peanuts peanuts actually small sized unsalted sure error vendor intended represent product jumbo\n",
      "\n",
      "Clean Review # 3\n",
      " delight  says it all\n",
      "confection around centuries light pillowy citrus gelatin nuts case filberts cut tiny squares liberally coated powdered sugar tiny mouthful heaven chewy flavorful highly recommend yummy treat familiar story c lewis lion witch wardrobe treat seduces edmund selling brother sisters witch\n",
      "\n",
      "Clean Review # 4\n",
      "cough medicine\n",
      "looking secret ingredient robitussin believe found got addition root beer extract ordered good made cherry soda flavor medicinal\n",
      "\n",
      "Clean Review # 5\n",
      "great taffy\n",
      "great taffy great price wide assortment yummy taffy delivery quick taffy lover deal\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Inspect the cleaned summaries and texts to ensure they have been cleaned well\n",
    "for i in range(5):\n",
    "    print(\"Clean Review #\",i+1)\n",
    "    print(clean_summaries[i])\n",
    "    print(clean_texts[i])\n",
    "    print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 207,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def count_words(count_dict, text):\n",
    "    '''Count the number of occurrences of each word in a set of text'''\n",
    "    for sentence in text:\n",
    "        for word in sentence.split():\n",
    "            if word not in count_dict:\n",
    "                count_dict[word] = 1\n",
    "            else:\n",
    "                count_dict[word] += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 208,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Size of Vocabulary: 132884\n"
     ]
    }
   ],
   "source": [
    "# Find the number of times each word was used and the size of the vocabulary\n",
    "word_counts = {}\n",
    "\n",
    "count_words(word_counts, clean_summaries)\n",
    "count_words(word_counts, clean_texts)\n",
    "            \n",
    "print(\"Size of Vocabulary:\", len(word_counts))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 209,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Word embeddings: 484557\n"
     ]
    }
   ],
   "source": [
    "# Load Conceptnet Numberbatch's (CN) embeddings, similar to GloVe, but probably better \n",
    "# (https://github.com/commonsense/conceptnet-numberbatch)\n",
    "embeddings_index = {}\n",
    "with open('/Users/Dave/Desktop/Programming/numberbatch-en-17.02.txt', encoding='utf-8') as f:\n",
    "    for line in f:\n",
    "        values = line.split(' ')\n",
    "        word = values[0]\n",
    "        embedding = np.asarray(values[1:], dtype='float32')\n",
    "        embeddings_index[word] = embedding\n",
    "\n",
    "print('Word embeddings:', len(embeddings_index))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 210,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of words missing from CN: 3044\n",
      "Percent of words that are missing from vocabulary: 2.29%\n"
     ]
    }
   ],
   "source": [
    "# Find the number of words that are missing from CN, and are used more than our threshold.\n",
    "missing_words = 0\n",
    "threshold = 20\n",
    "\n",
    "for word, count in word_counts.items():\n",
    "    if count > threshold:\n",
    "        if word not in embeddings_index:\n",
    "            missing_words += 1\n",
    "            \n",
    "missing_ratio = round(missing_words/len(word_counts),4)*100\n",
    "            \n",
    "print(\"Number of words missing from CN:\", missing_words)\n",
    "print(\"Percent of words that are missing from vocabulary: {}%\".format(missing_ratio))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I use a threshold of 20, so that words not in CN can be added to our word_embedding_matrix, but they need to be common enough in the reviews so that the model can understand their meaning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 211,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of unique words: 132884\n",
      "Number of words we will use: 65469\n",
      "Percent of words we will use: 49.27%\n"
     ]
    }
   ],
   "source": [
    "# Limit the vocab that we will use to words that appear ≥ threshold or are in GloVe\n",
    "\n",
    "#dictionary to convert words to integers\n",
    "vocab_to_int = {} \n",
    "\n",
    "value = 0\n",
    "for word, count in word_counts.items():\n",
    "    if count >= threshold or word in embeddings_index:\n",
    "        vocab_to_int[word] = value\n",
    "        value += 1\n",
    "\n",
    "# Special tokens that will be added to our vocab\n",
    "codes = [\"<UNK>\",\"<PAD>\",\"<EOS>\",\"<GO>\"]   \n",
    "\n",
    "# Add codes to vocab\n",
    "for code in codes:\n",
    "    vocab_to_int[code] = len(vocab_to_int)\n",
    "\n",
    "# Dictionary to convert integers to words\n",
    "int_to_vocab = {}\n",
    "for word, value in vocab_to_int.items():\n",
    "    int_to_vocab[value] = word\n",
    "\n",
    "usage_ratio = round(len(vocab_to_int) / len(word_counts),4)*100\n",
    "\n",
    "print(\"Total number of unique words:\", len(word_counts))\n",
    "print(\"Number of words we will use:\", len(vocab_to_int))\n",
    "print(\"Percent of words we will use: {}%\".format(usage_ratio))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 212,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "65469\n"
     ]
    }
   ],
   "source": [
    "# Need to use 300 for embedding dimensions to match CN's vectors.\n",
    "embedding_dim = 300\n",
    "nb_words = len(vocab_to_int)\n",
    "\n",
    "# Create matrix with default values of zero\n",
    "word_embedding_matrix = np.zeros((nb_words, embedding_dim), dtype=np.float32)\n",
    "for word, i in vocab_to_int.items():\n",
    "    if word in embeddings_index:\n",
    "        word_embedding_matrix[i] = embeddings_index[word]\n",
    "    else:\n",
    "        # If word not in CN, create a random embedding for it\n",
    "        new_embedding = np.array(np.random.uniform(-1.0, 1.0, embedding_dim))\n",
    "        embeddings_index[word] = new_embedding\n",
    "        word_embedding_matrix[i] = new_embedding\n",
    "\n",
    "# Check if value matches len(vocab_to_int)\n",
    "print(len(word_embedding_matrix))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 213,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def convert_to_ints(text, word_count, unk_count, eos=False):\n",
    "    '''Convert words in text to an integer.\n",
    "       If word is not in vocab_to_int, use UNK's integer.\n",
    "       Total the number of words and UNKs.\n",
    "       Add EOS token to the end of texts'''\n",
    "    ints = []\n",
    "    for sentence in text:\n",
    "        sentence_ints = []\n",
    "        for word in sentence.split():\n",
    "            word_count += 1\n",
    "            if word in vocab_to_int:\n",
    "                sentence_ints.append(vocab_to_int[word])\n",
    "            else:\n",
    "                sentence_ints.append(vocab_to_int[\"<UNK>\"])\n",
    "                unk_count += 1\n",
    "        if eos:\n",
    "            sentence_ints.append(vocab_to_int[\"<EOS>\"])\n",
    "        ints.append(sentence_ints)\n",
    "    return ints, word_count, unk_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 214,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of words in headlines: 25679946\n",
      "Total number of UNKs in headlines: 170450\n",
      "Percent of words that are UNK: 0.66%\n"
     ]
    }
   ],
   "source": [
    "# Apply convert_to_ints to clean_summaries and clean_texts\n",
    "word_count = 0\n",
    "unk_count = 0\n",
    "\n",
    "int_summaries, word_count, unk_count = convert_to_ints(clean_summaries, word_count, unk_count)\n",
    "int_texts, word_count, unk_count = convert_to_ints(clean_texts, word_count, unk_count, eos=True)\n",
    "\n",
    "unk_percent = round(unk_count/word_count,4)*100\n",
    "\n",
    "print(\"Total number of words in headlines:\", word_count)\n",
    "print(\"Total number of UNKs in headlines:\", unk_count)\n",
    "print(\"Percent of words that are UNK: {}%\".format(unk_percent))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 215,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def create_lengths(text):\n",
    "    '''Create a data frame of the sentence lengths from a text'''\n",
    "    lengths = []\n",
    "    for sentence in text:\n",
    "        lengths.append(len(sentence))\n",
    "    return pd.DataFrame(lengths, columns=['counts'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 216,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Summaries:\n",
      "              counts\n",
      "count  568412.000000\n",
      "mean        4.181620\n",
      "std         2.657872\n",
      "min         0.000000\n",
      "25%         2.000000\n",
      "50%         4.000000\n",
      "75%         5.000000\n",
      "max        48.000000\n",
      "\n",
      "Texts:\n",
      "              counts\n",
      "count  568412.000000\n",
      "mean       41.996782\n",
      "std        42.520854\n",
      "min         1.000000\n",
      "25%        18.000000\n",
      "50%        29.000000\n",
      "75%        50.000000\n",
      "max      2085.000000\n"
     ]
    }
   ],
   "source": [
    "lengths_summaries = create_lengths(int_summaries)\n",
    "lengths_texts = create_lengths(int_texts)\n",
    "\n",
    "print(\"Summaries:\")\n",
    "print(lengths_summaries.describe())\n",
    "print()\n",
    "print(\"Texts:\")\n",
    "print(lengths_texts.describe())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 217,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "84.0\n",
      "115.0\n",
      "207.0\n"
     ]
    }
   ],
   "source": [
    "# Inspect the length of texts\n",
    "print(np.percentile(lengths_texts.counts, 90))\n",
    "print(np.percentile(lengths_texts.counts, 95))\n",
    "print(np.percentile(lengths_texts.counts, 99))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 218,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "8.0\n",
      "9.0\n",
      "13.0\n"
     ]
    }
   ],
   "source": [
    "# Inspect the length of summaries\n",
    "print(np.percentile(lengths_summaries.counts, 90))\n",
    "print(np.percentile(lengths_summaries.counts, 95))\n",
    "print(np.percentile(lengths_summaries.counts, 99))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 219,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def unk_counter(sentence):\n",
    "    '''Counts the number of time UNK appears in a sentence.'''\n",
    "    unk_count = 0\n",
    "    for word in sentence:\n",
    "        if word == vocab_to_int[\"<UNK>\"]:\n",
    "            unk_count += 1\n",
    "    return unk_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 220,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "429210\n",
      "429210\n"
     ]
    }
   ],
   "source": [
    "# Sort the summaries and texts by the length of the texts, shortest to longest\n",
    "# Limit the length of summaries and texts based on the min and max ranges.\n",
    "# Remove reviews that include too many UNKs\n",
    "\n",
    "sorted_summaries = []\n",
    "sorted_texts = []\n",
    "max_text_length = 84\n",
    "max_summary_length = 13\n",
    "min_length = 2\n",
    "unk_text_limit = 1\n",
    "unk_summary_limit = 0\n",
    "\n",
    "for length in range(min(lengths_texts.counts), max_text_length): \n",
    "    for count, words in enumerate(int_summaries):\n",
    "        if (len(int_summaries[count]) >= min_length and\n",
    "            len(int_summaries[count]) <= max_summary_length and\n",
    "            len(int_texts[count]) >= min_length and\n",
    "            unk_counter(int_summaries[count]) <= unk_summary_limit and\n",
    "            unk_counter(int_texts[count]) <= unk_text_limit and\n",
    "            length == len(int_texts[count])\n",
    "           ):\n",
    "            sorted_summaries.append(int_summaries[count])\n",
    "            sorted_texts.append(int_texts[count])\n",
    "        \n",
    "# Compare lengths to ensure they match\n",
    "print(len(sorted_summaries))\n",
    "print(len(sorted_texts))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building the Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 221,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def model_inputs():\n",
    "    '''Create palceholders for inputs to the model'''\n",
    "    \n",
    "    input_data = tf.placeholder(tf.int32, [None, None], name='input')\n",
    "    targets = tf.placeholder(tf.int32, [None, None], name='targets')\n",
    "    lr = tf.placeholder(tf.float32, name='learning_rate')\n",
    "    keep_prob = tf.placeholder(tf.float32, name='keep_prob')\n",
    "    summary_length = tf.placeholder(tf.int32, (None,), name='summary_length')\n",
    "    max_summary_length = tf.reduce_max(summary_length, name='max_dec_len')\n",
    "    text_length = tf.placeholder(tf.int32, (None,), name='text_length')\n",
    "\n",
    "    return input_data, targets, lr, keep_prob, summary_length, max_summary_length, text_length"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 222,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def process_encoding_input(target_data, vocab_to_int, batch_size):\n",
    "    '''Remove the last word id from each batch and concat the <GO> to the begining of each batch'''\n",
    "    \n",
    "    ending = tf.strided_slice(target_data, [0, 0], [batch_size, -1], [1, 1])\n",
    "    dec_input = tf.concat([tf.fill([batch_size, 1], vocab_to_int['<GO>']), ending], 1)\n",
    "\n",
    "    return dec_input"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 223,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def encoding_layer(rnn_size, sequence_length, num_layers, rnn_inputs, keep_prob):\n",
    "    '''Create the encoding layer'''\n",
    "    \n",
    "    for layer in range(num_layers):\n",
    "        with tf.variable_scope('encoder_{}'.format(layer)):\n",
    "            cell_fw = tf.contrib.rnn.LSTMCell(rnn_size,\n",
    "                                              initializer=tf.random_uniform_initializer(-0.1, 0.1, seed=2))\n",
    "            cell_fw = tf.contrib.rnn.DropoutWrapper(cell_fw, \n",
    "                                                    input_keep_prob = keep_prob)\n",
    "\n",
    "            cell_bw = tf.contrib.rnn.LSTMCell(rnn_size,\n",
    "                                              initializer=tf.random_uniform_initializer(-0.1, 0.1, seed=2))\n",
    "            cell_bw = tf.contrib.rnn.DropoutWrapper(cell_bw, \n",
    "                                                    input_keep_prob = keep_prob)\n",
    "\n",
    "            enc_output, enc_state = tf.nn.bidirectional_dynamic_rnn(cell_fw, \n",
    "                                                                    cell_bw, \n",
    "                                                                    rnn_inputs,\n",
    "                                                                    sequence_length,\n",
    "                                                                    dtype=tf.float32)\n",
    "    # Join outputs since we are using a bidirectional RNN\n",
    "    enc_output = tf.concat(enc_output,2)\n",
    "    \n",
    "    return enc_output, enc_state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 224,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def training_decoding_layer(dec_embed_input, summary_length, dec_cell, initial_state, output_layer, \n",
    "                            vocab_size, max_summary_length):\n",
    "    '''Create the training logits'''\n",
    "    \n",
    "    training_helper = tf.contrib.seq2seq.TrainingHelper(inputs=dec_embed_input,\n",
    "                                                        sequence_length=summary_length,\n",
    "                                                        time_major=False)\n",
    "\n",
    "    training_decoder = tf.contrib.seq2seq.BasicDecoder(dec_cell,\n",
    "                                                       training_helper,\n",
    "                                                       initial_state,\n",
    "                                                       output_layer) \n",
    "\n",
    "    training_logits, _ = tf.contrib.seq2seq.dynamic_decode(training_decoder,\n",
    "                                                           output_time_major=False,\n",
    "                                                           impute_finished=True,\n",
    "                                                           maximum_iterations=max_summary_length)\n",
    "    return training_logits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 225,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def inference_decoding_layer(embeddings, start_token, end_token, dec_cell, initial_state, output_layer,\n",
    "                             max_summary_length, batch_size):\n",
    "    '''Create the inference logits'''\n",
    "    \n",
    "    start_tokens = tf.tile(tf.constant([start_token], dtype=tf.int32), [batch_size], name='start_tokens')\n",
    "    \n",
    "    inference_helper = tf.contrib.seq2seq.GreedyEmbeddingHelper(embeddings,\n",
    "                                                                start_tokens,\n",
    "                                                                end_token)\n",
    "                \n",
    "    inference_decoder = tf.contrib.seq2seq.BasicDecoder(dec_cell,\n",
    "                                                        inference_helper,\n",
    "                                                        initial_state,\n",
    "                                                        output_layer)\n",
    "                \n",
    "    inference_logits, _ = tf.contrib.seq2seq.dynamic_decode(inference_decoder,\n",
    "                                                            output_time_major=False,\n",
    "                                                            impute_finished=True,\n",
    "                                                            maximum_iterations=max_summary_length)\n",
    "    \n",
    "    return inference_logits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 226,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def decoding_layer(dec_embed_input, embeddings, enc_output, enc_state, vocab_size, text_length, summary_length, \n",
    "                   max_summary_length, rnn_size, vocab_to_int, keep_prob, batch_size, num_layers):\n",
    "    '''Create the decoding cell and attention for the training and inference decoding layers'''\n",
    "    \n",
    "    for layer in range(num_layers):\n",
    "        with tf.variable_scope('decoder_{}'.format(layer)):\n",
    "            lstm = tf.contrib.rnn.LSTMCell(rnn_size,\n",
    "                                           initializer=tf.random_uniform_initializer(-0.1, 0.1, seed=2))\n",
    "            dec_cell = tf.contrib.rnn.DropoutWrapper(lstm, \n",
    "                                                     input_keep_prob = keep_prob)\n",
    "    \n",
    "    output_layer = Dense(vocab_size,\n",
    "                         kernel_initializer = tf.truncated_normal_initializer(mean = 0.0, stddev=0.1))\n",
    "    \n",
    "    attn_mech = tf.contrib.seq2seq.BahdanauAttention(rnn_size,\n",
    "                                                  enc_output,\n",
    "                                                  text_length,\n",
    "                                                  normalize=False,\n",
    "                                                  name='BahdanauAttention')\n",
    "\n",
    "    dec_cell = tf.contrib.seq2seq.DynamicAttentionWrapper(dec_cell,\n",
    "                                                          attn_mech,\n",
    "                                                          rnn_size)\n",
    "            \n",
    "    initial_state = tf.contrib.seq2seq.DynamicAttentionWrapperState(enc_state[0],\n",
    "                                                                    _zero_state_tensors(rnn_size, \n",
    "                                                                                        batch_size, \n",
    "                                                                                        tf.float32)) \n",
    "    with tf.variable_scope(\"decode\"):\n",
    "        training_logits = training_decoding_layer(dec_embed_input, \n",
    "                                                  summary_length, \n",
    "                                                  dec_cell, \n",
    "                                                  initial_state,\n",
    "                                                  output_layer,\n",
    "                                                  vocab_size, \n",
    "                                                  max_summary_length)\n",
    "    with tf.variable_scope(\"decode\", reuse=True):\n",
    "        inference_logits = inference_decoding_layer(embeddings,  \n",
    "                                                    vocab_to_int['<GO>'], \n",
    "                                                    vocab_to_int['<EOS>'],\n",
    "                                                    dec_cell, \n",
    "                                                    initial_state, \n",
    "                                                    output_layer,\n",
    "                                                    max_summary_length,\n",
    "                                                    batch_size)\n",
    "\n",
    "    return training_logits, inference_logits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 227,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def seq2seq_model(input_data, target_data, keep_prob, text_length, summary_length, max_summary_length, \n",
    "                  vocab_size, rnn_size, num_layers, vocab_to_int, batch_size):\n",
    "    '''Use the previous functions to create the training and inference logits'''\n",
    "    \n",
    "    # Use Numberbatch's embeddings and the newly created ones as our embeddings\n",
    "    embeddings = word_embedding_matrix\n",
    "    \n",
    "    enc_embed_input = tf.nn.embedding_lookup(embeddings, input_data)\n",
    "    enc_output, enc_state = encoding_layer(rnn_size, text_length, num_layers, enc_embed_input, keep_prob)\n",
    "    \n",
    "    dec_input = process_encoding_input(target_data, vocab_to_int, batch_size)\n",
    "    dec_embed_input = tf.nn.embedding_lookup(embeddings, dec_input)\n",
    "    \n",
    "    training_logits, inference_logits  = decoding_layer(dec_embed_input, \n",
    "                                                        embeddings,\n",
    "                                                        enc_output,\n",
    "                                                        enc_state, \n",
    "                                                        vocab_size, \n",
    "                                                        text_length, \n",
    "                                                        summary_length, \n",
    "                                                        max_summary_length,\n",
    "                                                        rnn_size, \n",
    "                                                        vocab_to_int, \n",
    "                                                        keep_prob, \n",
    "                                                        batch_size,\n",
    "                                                        num_layers)\n",
    "    \n",
    "    return training_logits, inference_logits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 228,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def pad_sentence_batch(sentence_batch):\n",
    "    \"\"\"Pad sentences with <PAD> so that each sentence of a batch has the same length\"\"\"\n",
    "    max_sentence = max([len(sentence) for sentence in sentence_batch])\n",
    "    return [sentence + [vocab_to_int['<PAD>']] * (max_sentence - len(sentence)) for sentence in sentence_batch]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 229,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_batches(summaries, texts, batch_size):\n",
    "    \"\"\"Batch summaries, texts, and the lengths of their sentences together\"\"\"\n",
    "    for batch_i in range(0, len(texts)//batch_size):\n",
    "        start_i = batch_i * batch_size\n",
    "        summaries_batch = summaries[start_i:start_i + batch_size]\n",
    "        texts_batch = texts[start_i:start_i + batch_size]\n",
    "        pad_summaries_batch = np.array(pad_sentence_batch(summaries_batch))\n",
    "        pad_texts_batch = np.array(pad_sentence_batch(texts_batch))\n",
    "        \n",
    "        # Need the lengths for the _lengths parameters\n",
    "        pad_summaries_lengths = []\n",
    "        for summary in pad_summaries_batch:\n",
    "            pad_summaries_lengths.append(len(summary))\n",
    "        \n",
    "        pad_texts_lengths = []\n",
    "        for text in pad_texts_batch:\n",
    "            pad_texts_lengths.append(len(text))\n",
    "        \n",
    "        yield pad_summaries_batch, pad_texts_batch, pad_summaries_lengths, pad_texts_lengths"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 230,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Set the Hyperparameters\n",
    "epochs = 100\n",
    "batch_size = 64\n",
    "rnn_size = 256\n",
    "num_layers = 2\n",
    "learning_rate = 0.005\n",
    "keep_probability = 0.75"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 231,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Graph is built.\n"
     ]
    }
   ],
   "source": [
    "# Build the graph\n",
    "train_graph = tf.Graph()\n",
    "# Set the graph to default to ensure that it is ready for training\n",
    "with train_graph.as_default():\n",
    "    \n",
    "    # Load the model inputs    \n",
    "    input_data, targets, lr, keep_prob, summary_length, max_summary_length, text_length = model_inputs()\n",
    "\n",
    "    # Create the training and inference logits\n",
    "    training_logits, inference_logits = seq2seq_model(tf.reverse(input_data, [-1]),\n",
    "                                                      targets, \n",
    "                                                      keep_prob,   \n",
    "                                                      text_length,\n",
    "                                                      summary_length,\n",
    "                                                      max_summary_length,\n",
    "                                                      len(vocab_to_int)+1,\n",
    "                                                      rnn_size, \n",
    "                                                      num_layers, \n",
    "                                                      vocab_to_int,\n",
    "                                                      batch_size)\n",
    "    \n",
    "    # Create tensors for the training logits and inference logits\n",
    "    training_logits = tf.identity(training_logits.rnn_output, 'logits')\n",
    "    inference_logits = tf.identity(inference_logits.sample_id, name='predictions')\n",
    "    \n",
    "    # Create the weights for sequence_loss\n",
    "    masks = tf.sequence_mask(summary_length, max_summary_length, dtype=tf.float32, name='masks')\n",
    "\n",
    "    with tf.name_scope(\"optimization\"):\n",
    "        # Loss function\n",
    "        cost = tf.contrib.seq2seq.sequence_loss(\n",
    "            training_logits,\n",
    "            targets,\n",
    "            masks)\n",
    "\n",
    "        # Optimizer\n",
    "        optimizer = tf.train.AdamOptimizer(learning_rate)\n",
    "\n",
    "        # Gradient Clipping\n",
    "        gradients = optimizer.compute_gradients(cost)\n",
    "        capped_gradients = [(tf.clip_by_value(grad, -5., 5.), var) for grad, var in gradients if grad is not None]\n",
    "        train_op = optimizer.apply_gradients(capped_gradients)\n",
    "print(\"Graph is built.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training the Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since I am training this model on my MacBook Pro, it would take me days if I used the whole dataset. For this reason, I am only going to use a subset of the data, so that I can train it over night. Normally I use [FloydHub's](https://www.floydhub.com/) services for my GPU needs, but it would take quite a bit of time to upload the dataset and ConceptNet Numberbatch, so I'm not going to bother with that for this project.\n",
    "\n",
    "I chose not use use the start of the subset because I didn't want to make it too easy for my model. The texts that I am using are closer to the median lengths; I thought this would be more fair."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 234,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The shortest text length: 25\n",
      "The longest text length: 31\n"
     ]
    }
   ],
   "source": [
    "# Subset the data for training\n",
    "start = 200000\n",
    "end = start + 50000\n",
    "sorted_summaries_short = sorted_summaries[start:end]\n",
    "sorted_texts_short = sorted_texts[start:end]\n",
    "print(\"The shortest text length:\", len(sorted_texts_short[0]))\n",
    "print(\"The longest text length:\",len(sorted_texts_short[-1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch   1/100 Batch   20/781 - Loss:  4.470, Seconds: 156.00\n",
      "Epoch   1/100 Batch   40/781 - Loss:  2.863, Seconds: 105.20\n",
      "Epoch   1/100 Batch   60/781 - Loss:  2.652, Seconds: 151.58\n",
      "Epoch   1/100 Batch   80/781 - Loss:  2.736, Seconds: 117.19\n",
      "Epoch   1/100 Batch  100/781 - Loss:  2.686, Seconds: 118.42\n",
      "Epoch   1/100 Batch  120/781 - Loss:  2.423, Seconds: 140.21\n",
      "Epoch   1/100 Batch  140/781 - Loss:  2.696, Seconds: 152.89\n",
      "Epoch   1/100 Batch  160/781 - Loss:  2.606, Seconds: 128.19\n",
      "Epoch   1/100 Batch  180/781 - Loss:  2.525, Seconds: 151.52\n",
      "Epoch   1/100 Batch  200/781 - Loss:  2.597, Seconds: 140.84\n",
      "Epoch   1/100 Batch  220/781 - Loss:  2.515, Seconds: 130.87\n",
      "Epoch   1/100 Batch  240/781 - Loss:  2.402, Seconds: 131.02\n",
      "Average loss for this update: 2.734\n",
      "New Record!\n",
      "Epoch   1/100 Batch  260/781 - Loss:  2.382, Seconds: 106.18\n",
      "Epoch   1/100 Batch  280/781 - Loss:  2.354, Seconds: 124.90\n",
      "Epoch   1/100 Batch  300/781 - Loss:  2.306, Seconds: 148.73\n",
      "Epoch   1/100 Batch  320/781 - Loss:  2.637, Seconds: 142.09\n",
      "Epoch   1/100 Batch  340/781 - Loss:  2.680, Seconds: 140.91\n",
      "Epoch   1/100 Batch  360/781 - Loss:  2.559, Seconds: 96.81\n",
      "Epoch   1/100 Batch  380/781 - Loss:  2.448, Seconds: 130.00\n",
      "Epoch   1/100 Batch  400/781 - Loss:  2.615, Seconds: 108.58\n",
      "Epoch   1/100 Batch  420/781 - Loss:  2.193, Seconds: 124.55\n",
      "Epoch   1/100 Batch  440/781 - Loss:  2.315, Seconds: 131.78\n",
      "Epoch   1/100 Batch  460/781 - Loss:  2.276, Seconds: 131.26\n",
      "Epoch   1/100 Batch  480/781 - Loss:  2.391, Seconds: 88.43\n",
      "Epoch   1/100 Batch  500/781 - Loss:  2.455, Seconds: 120.87\n",
      "Average loss for this update: 2.436\n",
      "New Record!\n",
      "Epoch   1/100 Batch  520/781 - Loss:  2.459, Seconds: 94.69\n",
      "Epoch   1/100 Batch  540/781 - Loss:  2.412, Seconds: 108.36\n",
      "Epoch   1/100 Batch  560/781 - Loss:  2.320, Seconds: 128.48\n",
      "Epoch   1/100 Batch  580/781 - Loss:  2.195, Seconds: 121.47\n",
      "Epoch   1/100 Batch  600/781 - Loss:  2.353, Seconds: 120.64\n",
      "Epoch   1/100 Batch  620/781 - Loss:  2.079, Seconds: 153.50\n",
      "Epoch   1/100 Batch  640/781 - Loss:  2.254, Seconds: 154.52\n",
      "Epoch   1/100 Batch  660/781 - Loss:  2.570, Seconds: 124.24\n",
      "Epoch   1/100 Batch  680/781 - Loss:  2.352, Seconds: 154.14\n",
      "Epoch   1/100 Batch  700/781 - Loss:  2.170, Seconds: 121.13\n",
      "Epoch   1/100 Batch  720/781 - Loss:  2.159, Seconds: 121.35\n",
      "Epoch   1/100 Batch  740/781 - Loss:  2.201, Seconds: 121.36\n",
      "Epoch   1/100 Batch  760/781 - Loss:  2.037, Seconds: 122.65\n",
      "Average loss for this update: 2.253\n",
      "New Record!\n",
      "Epoch   1/100 Batch  780/781 - Loss:  2.088, Seconds: 136.48\n",
      "Epoch   2/100 Batch   20/781 - Loss:  2.298, Seconds: 152.02\n",
      "Epoch   2/100 Batch   40/781 - Loss:  2.171, Seconds: 108.29\n",
      "Epoch   2/100 Batch   60/781 - Loss:  2.015, Seconds: 152.19\n",
      "Epoch   2/100 Batch   80/781 - Loss:  2.052, Seconds: 117.14\n",
      "Epoch   2/100 Batch  100/781 - Loss:  2.069, Seconds: 119.78\n",
      "Epoch   2/100 Batch  120/781 - Loss:  1.831, Seconds: 145.87\n",
      "Epoch   2/100 Batch  140/781 - Loss:  2.136, Seconds: 151.15\n",
      "Epoch   2/100 Batch  160/781 - Loss:  2.082, Seconds: 132.47\n",
      "Epoch   2/100 Batch  180/781 - Loss:  2.019, Seconds: 153.08\n",
      "Epoch   2/100 Batch  200/781 - Loss:  2.052, Seconds: 142.47\n",
      "Epoch   2/100 Batch  220/781 - Loss:  1.969, Seconds: 130.85\n",
      "Epoch   2/100 Batch  240/781 - Loss:  1.819, Seconds: 129.27\n",
      "Average loss for this update: 2.028\n",
      "New Record!\n",
      "Epoch   2/100 Batch  260/781 - Loss:  1.869, Seconds: 103.74\n",
      "Epoch   2/100 Batch  280/781 - Loss:  1.829, Seconds: 130.26\n",
      "Epoch   2/100 Batch  300/781 - Loss:  1.803, Seconds: 151.96\n",
      "Epoch   2/100 Batch  320/781 - Loss:  2.139, Seconds: 141.95\n",
      "Epoch   2/100 Batch  340/781 - Loss:  2.189, Seconds: 145.17\n",
      "Epoch   2/100 Batch  360/781 - Loss:  2.047, Seconds: 98.12\n",
      "Epoch   2/100 Batch  380/781 - Loss:  1.955, Seconds: 130.35\n",
      "Epoch   2/100 Batch  400/781 - Loss:  2.080, Seconds: 108.95\n",
      "Epoch   2/100 Batch  420/781 - Loss:  1.717, Seconds: 120.29\n",
      "Epoch   2/100 Batch  440/781 - Loss:  1.842, Seconds: 130.07\n",
      "Epoch   2/100 Batch  460/781 - Loss:  1.798, Seconds: 132.52\n",
      "Epoch   2/100 Batch  480/781 - Loss:  1.929, Seconds: 87.39\n",
      "Epoch   2/100 Batch  500/781 - Loss:  2.050, Seconds: 122.04\n",
      "Average loss for this update: 1.953\n",
      "New Record!\n",
      "Epoch   2/100 Batch  520/781 - Loss:  2.037, Seconds: 94.28\n",
      "Epoch   2/100 Batch  540/781 - Loss:  1.972, Seconds: 109.17\n",
      "Epoch   2/100 Batch  560/781 - Loss:  1.923, Seconds: 131.12\n",
      "Epoch   2/100 Batch  580/781 - Loss:  1.728, Seconds: 120.70\n",
      "Epoch   2/100 Batch  600/781 - Loss:  1.894, Seconds: 120.67\n",
      "Epoch   2/100 Batch  620/781 - Loss:  1.708, Seconds: 154.05\n",
      "Epoch   2/100 Batch  640/781 - Loss:  1.859, Seconds: 153.93\n",
      "Epoch   2/100 Batch  660/781 - Loss:  2.167, Seconds: 121.69\n",
      "Epoch   2/100 Batch  680/781 - Loss:  1.982, Seconds: 154.94\n",
      "Epoch   2/100 Batch  700/781 - Loss:  1.804, Seconds: 122.34\n",
      "Epoch   2/100 Batch  720/781 - Loss:  1.800, Seconds: 120.82\n",
      "Epoch   2/100 Batch  740/781 - Loss:  1.778, Seconds: 121.45\n",
      "Epoch   2/100 Batch  760/781 - Loss:  1.712, Seconds: 121.23\n",
      "Average loss for this update: 1.857\n",
      "New Record!\n",
      "Epoch   2/100 Batch  780/781 - Loss:  1.737, Seconds: 136.31\n",
      "Epoch   3/100 Batch   20/781 - Loss:  2.015, Seconds: 150.41\n",
      "Epoch   3/100 Batch   40/781 - Loss:  1.906, Seconds: 108.50\n",
      "Epoch   3/100 Batch   60/781 - Loss:  1.721, Seconds: 155.11\n",
      "Epoch   3/100 Batch   80/781 - Loss:  1.728, Seconds: 117.11\n",
      "Epoch   3/100 Batch  100/781 - Loss:  1.776, Seconds: 121.61\n",
      "Epoch   3/100 Batch  120/781 - Loss:  1.569, Seconds: 141.59\n",
      "Epoch   3/100 Batch  140/781 - Loss:  1.845, Seconds: 152.36\n",
      "Epoch   3/100 Batch  160/781 - Loss:  1.811, Seconds: 129.96\n",
      "Epoch   3/100 Batch  180/781 - Loss:  1.749, Seconds: 153.36\n",
      "Epoch   3/100 Batch  200/781 - Loss:  1.770, Seconds: 139.69\n",
      "Epoch   3/100 Batch  220/781 - Loss:  1.698, Seconds: 133.17\n",
      "Epoch   3/100 Batch  240/781 - Loss:  1.535, Seconds: 130.97\n",
      "Average loss for this update: 1.746\n",
      "New Record!\n",
      "Epoch   3/100 Batch  260/781 - Loss:  1.594, Seconds: 103.10\n",
      "Epoch   3/100 Batch  280/781 - Loss:  1.549, Seconds: 130.66\n",
      "Epoch   3/100 Batch  300/781 - Loss:  1.524, Seconds: 151.88\n",
      "Epoch   3/100 Batch  320/781 - Loss:  1.861, Seconds: 142.28\n",
      "Epoch   3/100 Batch  340/781 - Loss:  1.893, Seconds: 139.69\n",
      "Epoch   3/100 Batch  360/781 - Loss:  1.769, Seconds: 98.39\n",
      "Epoch   3/100 Batch  380/781 - Loss:  1.689, Seconds: 131.38\n",
      "Epoch   3/100 Batch  400/781 - Loss:  1.803, Seconds: 109.51\n",
      "Epoch   3/100 Batch  420/781 - Loss:  1.455, Seconds: 122.05\n",
      "Epoch   3/100 Batch  440/781 - Loss:  1.588, Seconds: 132.11\n",
      "Epoch   3/100 Batch  460/781 - Loss:  1.540, Seconds: 130.48\n",
      "Epoch   3/100 Batch  480/781 - Loss:  1.687, Seconds: 88.24\n",
      "Epoch   3/100 Batch  500/781 - Loss:  1.818, Seconds: 121.00\n",
      "Average loss for this update: 1.688\n",
      "New Record!\n",
      "Epoch   3/100 Batch  520/781 - Loss:  1.787, Seconds: 94.69\n",
      "Epoch   3/100 Batch  540/781 - Loss:  1.689, Seconds: 109.86\n",
      "Epoch   3/100 Batch  560/781 - Loss:  1.678, Seconds: 131.70\n",
      "Epoch   3/100 Batch  580/781 - Loss:  1.488, Seconds: 120.36\n",
      "Epoch   3/100 Batch  600/781 - Loss:  1.637, Seconds: 122.60\n",
      "Epoch   3/100 Batch  620/781 - Loss:  1.493, Seconds: 153.82\n",
      "Epoch   3/100 Batch  640/781 - Loss:  1.627, Seconds: 158.39\n",
      "Epoch   3/100 Batch  660/781 - Loss:  1.910, Seconds: 121.86\n",
      "Epoch   3/100 Batch  680/781 - Loss:  1.755, Seconds: 153.93\n",
      "Epoch   3/100 Batch  700/781 - Loss:  1.590, Seconds: 121.65\n",
      "Epoch   3/100 Batch  720/781 - Loss:  1.585, Seconds: 122.38\n",
      "Epoch   3/100 Batch  740/781 - Loss:  1.540, Seconds: 122.20\n",
      "Epoch   3/100 Batch  760/781 - Loss:  1.515, Seconds: 122.62\n",
      "Average loss for this update: 1.623\n",
      "New Record!\n",
      "Epoch   3/100 Batch  780/781 - Loss:  1.515, Seconds: 135.59\n",
      "Epoch   4/100 Batch   20/781 - Loss:  1.793, Seconds: 152.96\n",
      "Epoch   4/100 Batch   40/781 - Loss:  1.713, Seconds: 107.82\n",
      "Epoch   4/100 Batch   60/781 - Loss:  1.527, Seconds: 153.71\n",
      "Epoch   4/100 Batch   80/781 - Loss:  1.513, Seconds: 118.78\n",
      "Epoch   4/100 Batch  100/781 - Loss:  1.577, Seconds: 121.06\n",
      "Epoch   4/100 Batch  120/781 - Loss:  1.397, Seconds: 143.41\n",
      "Epoch   4/100 Batch  140/781 - Loss:  1.635, Seconds: 152.24\n",
      "Epoch   4/100 Batch  160/781 - Loss:  1.616, Seconds: 131.96\n",
      "Epoch   4/100 Batch  180/781 - Loss:  1.570, Seconds: 152.69\n",
      "Epoch   4/100 Batch  200/781 - Loss:  1.580, Seconds: 141.25\n",
      "Epoch   4/100 Batch  220/781 - Loss:  1.510, Seconds: 131.03\n",
      "Epoch   4/100 Batch  240/781 - Loss:  1.337, Seconds: 129.62\n",
      "Average loss for this update: 1.551\n",
      "New Record!\n",
      "Epoch   4/100 Batch  260/781 - Loss:  1.416, Seconds: 104.76\n",
      "Epoch   4/100 Batch  280/781 - Loss:  1.371, Seconds: 130.99\n",
      "Epoch   4/100 Batch  300/781 - Loss:  1.353, Seconds: 150.54\n",
      "Epoch   4/100 Batch  320/781 - Loss:  1.648, Seconds: 142.99\n",
      "Epoch   4/100 Batch  340/781 - Loss:  1.692, Seconds: 142.90\n",
      "Epoch   4/100 Batch  360/781 - Loss:  1.584, Seconds: 98.49\n",
      "Epoch   4/100 Batch  380/781 - Loss:  1.505, Seconds: 136.00\n",
      "Epoch   4/100 Batch  400/781 - Loss:  1.604, Seconds: 110.08\n",
      "Epoch   4/100 Batch  420/781 - Loss:  1.286, Seconds: 121.17\n",
      "Epoch   4/100 Batch  440/781 - Loss:  1.411, Seconds: 130.35\n",
      "Epoch   4/100 Batch  460/781 - Loss:  1.384, Seconds: 131.63\n",
      "Epoch   4/100 Batch  480/781 - Loss:  1.495, Seconds: 86.78\n",
      "Epoch   4/100 Batch  500/781 - Loss:  1.641, Seconds: 120.57\n",
      "Average loss for this update: 1.505\n",
      "New Record!\n",
      "Epoch   4/100 Batch  520/781 - Loss:  1.616, Seconds: 94.21\n",
      "Epoch   4/100 Batch  540/781 - Loss:  1.508, Seconds: 109.50\n",
      "Epoch   4/100 Batch  560/781 - Loss:  1.489, Seconds: 131.04\n",
      "Epoch   4/100 Batch  580/781 - Loss:  1.307, Seconds: 122.48\n",
      "Epoch   4/100 Batch  600/781 - Loss:  1.456, Seconds: 121.87\n",
      "Epoch   4/100 Batch  620/781 - Loss:  1.348, Seconds: 153.92\n",
      "Epoch   4/100 Batch  640/781 - Loss:  1.453, Seconds: 156.55\n",
      "Epoch   4/100 Batch  660/781 - Loss:  1.715, Seconds: 122.58\n",
      "Epoch   4/100 Batch  680/781 - Loss:  1.590, Seconds: 154.58\n",
      "Epoch   4/100 Batch  700/781 - Loss:  1.429, Seconds: 123.25\n",
      "Epoch   4/100 Batch  720/781 - Loss:  1.440, Seconds: 120.86\n",
      "Epoch   4/100 Batch  740/781 - Loss:  1.379, Seconds: 119.59\n",
      "Epoch   4/100 Batch  760/781 - Loss:  1.374, Seconds: 123.34\n",
      "Average loss for this update: 1.456\n",
      "New Record!\n",
      "Epoch   4/100 Batch  780/781 - Loss:  1.364, Seconds: 136.66\n",
      "Epoch   5/100 Batch   20/781 - Loss:  1.621, Seconds: 153.30\n",
      "Epoch   5/100 Batch   40/781 - Loss:  1.567, Seconds: 106.96\n",
      "Epoch   5/100 Batch   60/781 - Loss:  1.385, Seconds: 151.55\n",
      "Epoch   5/100 Batch   80/781 - Loss:  1.366, Seconds: 117.78\n",
      "Epoch   5/100 Batch  100/781 - Loss:  1.419, Seconds: 120.49\n",
      "Epoch   5/100 Batch  120/781 - Loss:  1.270, Seconds: 140.66\n",
      "Epoch   5/100 Batch  140/781 - Loss:  1.478, Seconds: 153.51\n",
      "Epoch   5/100 Batch  160/781 - Loss:  1.474, Seconds: 131.34\n",
      "Epoch   5/100 Batch  180/781 - Loss:  1.425, Seconds: 152.85\n",
      "Epoch   5/100 Batch  200/781 - Loss:  1.437, Seconds: 143.59\n",
      "Epoch   5/100 Batch  220/781 - Loss:  1.374, Seconds: 133.55\n",
      "Epoch   5/100 Batch  240/781 - Loss:  1.216, Seconds: 130.37\n",
      "Average loss for this update: 1.406\n",
      "New Record!\n",
      "Epoch   5/100 Batch  260/781 - Loss:  1.271, Seconds: 103.74\n",
      "Epoch   5/100 Batch  280/781 - Loss:  1.266, Seconds: 130.39\n",
      "Epoch   5/100 Batch  300/781 - Loss:  1.227, Seconds: 155.28\n",
      "Epoch   5/100 Batch  320/781 - Loss:  1.486, Seconds: 143.04\n",
      "Epoch   5/100 Batch  340/781 - Loss:  1.552, Seconds: 141.86\n",
      "Epoch   5/100 Batch  360/781 - Loss:  1.423, Seconds: 99.56\n",
      "Epoch   5/100 Batch  380/781 - Loss:  1.365, Seconds: 131.81\n",
      "Epoch   5/100 Batch  400/781 - Loss:  1.456, Seconds: 113.05\n",
      "Epoch   5/100 Batch  420/781 - Loss:  1.178, Seconds: 119.84\n",
      "Epoch   5/100 Batch  440/781 - Loss:  1.290, Seconds: 131.00\n",
      "Epoch   5/100 Batch  460/781 - Loss:  1.240, Seconds: 133.82\n",
      "Epoch   5/100 Batch  480/781 - Loss:  1.355, Seconds: 89.87\n",
      "Epoch   5/100 Batch  500/781 - Loss:  1.496, Seconds: 122.93\n",
      "Average loss for this update: 1.368\n",
      "New Record!\n",
      "Epoch   5/100 Batch  520/781 - Loss:  1.464, Seconds: 96.14\n",
      "Epoch   5/100 Batch  540/781 - Loss:  1.356, Seconds: 109.62\n",
      "Epoch   5/100 Batch  560/781 - Loss:  1.362, Seconds: 133.67\n",
      "Epoch   5/100 Batch  580/781 - Loss:  1.194, Seconds: 121.81\n",
      "Epoch   5/100 Batch  600/781 - Loss:  1.342, Seconds: 121.25\n",
      "Epoch   5/100 Batch  620/781 - Loss:  1.239, Seconds: 156.98\n",
      "Epoch   5/100 Batch  640/781 - Loss:  1.323, Seconds: 156.76\n",
      "Epoch   5/100 Batch  660/781 - Loss:  1.556, Seconds: 122.91\n",
      "Epoch   5/100 Batch  680/781 - Loss:  1.453, Seconds: 158.00\n",
      "Epoch   5/100 Batch  700/781 - Loss:  1.308, Seconds: 123.65\n",
      "Epoch   5/100 Batch  720/781 - Loss:  1.317, Seconds: 123.67\n",
      "Epoch   5/100 Batch  740/781 - Loss:  1.267, Seconds: 125.47\n",
      "Epoch   5/100 Batch  760/781 - Loss:  1.275, Seconds: 122.67\n",
      "Average loss for this update: 1.331\n",
      "New Record!\n",
      "Epoch   5/100 Batch  780/781 - Loss:  1.254, Seconds: 136.56\n",
      "Epoch   6/100 Batch   20/781 - Loss:  1.486, Seconds: 152.52\n",
      "Epoch   6/100 Batch   40/781 - Loss:  1.449, Seconds: 111.01\n",
      "Epoch   6/100 Batch   60/781 - Loss:  1.277, Seconds: 152.59\n",
      "Epoch   6/100 Batch   80/781 - Loss:  1.241, Seconds: 120.53\n",
      "Epoch   6/100 Batch  100/781 - Loss:  1.297, Seconds: 120.91\n",
      "Epoch   6/100 Batch  120/781 - Loss:  1.159, Seconds: 144.18\n",
      "Epoch   6/100 Batch  140/781 - Loss:  1.360, Seconds: 158.39\n",
      "Epoch   6/100 Batch  160/781 - Loss:  1.336, Seconds: 132.86\n",
      "Epoch   6/100 Batch  180/781 - Loss:  1.293, Seconds: 155.74\n",
      "Epoch   6/100 Batch  200/781 - Loss:  1.321, Seconds: 145.67\n",
      "Epoch   6/100 Batch  220/781 - Loss:  1.267, Seconds: 135.30\n",
      "Epoch   6/100 Batch  240/781 - Loss:  1.100, Seconds: 133.28\n",
      "Average loss for this update: 1.288\n",
      "New Record!\n",
      "Epoch   6/100 Batch  260/781 - Loss:  1.168, Seconds: 103.96\n",
      "Epoch   6/100 Batch  280/781 - Loss:  1.156, Seconds: 131.70\n",
      "Epoch   6/100 Batch  300/781 - Loss:  1.133, Seconds: 155.86\n",
      "Epoch   6/100 Batch  320/781 - Loss:  1.361, Seconds: 145.41\n",
      "Epoch   6/100 Batch  340/781 - Loss:  1.412, Seconds: 145.41\n",
      "Epoch   6/100 Batch  360/781 - Loss:  1.329, Seconds: 100.96\n",
      "Epoch   6/100 Batch  380/781 - Loss:  1.266, Seconds: 134.49\n",
      "Epoch   6/100 Batch  400/781 - Loss:  1.314, Seconds: 110.95\n",
      "Epoch   6/100 Batch  420/781 - Loss:  1.093, Seconds: 126.54\n",
      "Epoch   6/100 Batch  440/781 - Loss:  1.206, Seconds: 135.67\n",
      "Epoch   6/100 Batch  460/781 - Loss:  1.151, Seconds: 137.55\n",
      "Epoch   6/100 Batch  480/781 - Loss:  1.241, Seconds: 90.75\n",
      "Epoch   6/100 Batch  500/781 - Loss:  1.377, Seconds: 125.70\n",
      "Average loss for this update: 1.26\n",
      "New Record!\n",
      "Epoch   6/100 Batch  520/781 - Loss:  1.361, Seconds: 94.51\n",
      "Epoch   6/100 Batch  540/781 - Loss:  1.250, Seconds: 113.06\n",
      "Epoch   6/100 Batch  560/781 - Loss:  1.268, Seconds: 134.60\n",
      "Epoch   6/100 Batch  580/781 - Loss:  1.088, Seconds: 123.48\n",
      "Epoch   6/100 Batch  600/781 - Loss:  1.224, Seconds: 123.78\n",
      "Epoch   6/100 Batch  620/781 - Loss:  1.153, Seconds: 156.48\n",
      "Epoch   6/100 Batch  640/781 - Loss:  1.231, Seconds: 157.48\n",
      "Epoch   6/100 Batch  660/781 - Loss:  1.447, Seconds: 123.59\n",
      "Epoch   6/100 Batch  680/781 - Loss:  1.363, Seconds: 157.39\n",
      "Epoch   6/100 Batch  700/781 - Loss:  1.227, Seconds: 128.72\n",
      "Epoch   6/100 Batch  720/781 - Loss:  1.238, Seconds: 126.35\n",
      "Epoch   6/100 Batch  740/781 - Loss:  1.174, Seconds: 125.93\n",
      "Epoch   6/100 Batch  760/781 - Loss:  1.191, Seconds: 125.24\n",
      "Average loss for this update: 1.239\n",
      "New Record!\n",
      "Epoch   6/100 Batch  780/781 - Loss:  1.206, Seconds: 139.27\n",
      "Epoch   7/100 Batch   20/781 - Loss:  1.416, Seconds: 155.27\n",
      "Epoch   7/100 Batch   40/781 - Loss:  1.381, Seconds: 110.29\n",
      "Epoch   7/100 Batch   60/781 - Loss:  1.193, Seconds: 156.02\n",
      "Epoch   7/100 Batch   80/781 - Loss:  1.177, Seconds: 122.76\n",
      "Epoch   7/100 Batch  100/781 - Loss:  1.233, Seconds: 122.76\n",
      "Epoch   7/100 Batch  120/781 - Loss:  1.110, Seconds: 144.25\n",
      "Epoch   7/100 Batch  140/781 - Loss:  1.274, Seconds: 156.67\n",
      "Epoch   7/100 Batch  160/781 - Loss:  1.251, Seconds: 138.95\n",
      "Epoch   7/100 Batch  180/781 - Loss:  1.230, Seconds: 159.33\n",
      "Epoch   7/100 Batch  200/781 - Loss:  1.244, Seconds: 147.36\n",
      "Epoch   7/100 Batch  220/781 - Loss:  1.189, Seconds: 136.09\n",
      "Epoch   7/100 Batch  240/781 - Loss:  1.037, Seconds: 136.38\n",
      "Average loss for this update: 1.217\n",
      "New Record!\n",
      "Epoch   7/100 Batch  260/781 - Loss:  1.106, Seconds: 103.50\n",
      "Epoch   7/100 Batch  280/781 - Loss:  1.092, Seconds: 135.91\n",
      "Epoch   7/100 Batch  300/781 - Loss:  1.064, Seconds: 156.83\n",
      "Epoch   7/100 Batch  320/781 - Loss:  1.267, Seconds: 146.38\n",
      "Epoch   7/100 Batch  340/781 - Loss:  1.321, Seconds: 145.75\n",
      "Epoch   7/100 Batch  360/781 - Loss:  1.238, Seconds: 101.32\n",
      "Epoch   7/100 Batch  380/781 - Loss:  1.187, Seconds: 133.62\n",
      "Epoch   7/100 Batch  400/781 - Loss:  1.232, Seconds: 114.49\n",
      "Epoch   7/100 Batch  420/781 - Loss:  1.007, Seconds: 126.89\n",
      "Epoch   7/100 Batch  440/781 - Loss:  1.119, Seconds: 140.93\n",
      "Epoch   7/100 Batch  460/781 - Loss:  1.067, Seconds: 139.15\n",
      "Epoch   7/100 Batch  480/781 - Loss:  1.148, Seconds: 91.13\n",
      "Epoch   7/100 Batch  500/781 - Loss:  1.280, Seconds: 126.78\n",
      "Average loss for this update: 1.175\n",
      "New Record!\n",
      "Epoch   7/100 Batch  520/781 - Loss:  1.278, Seconds: 126.51\n",
      "Epoch   7/100 Batch  540/781 - Loss:  1.166, Seconds: 116.39\n",
      "Epoch   7/100 Batch  560/781 - Loss:  1.180, Seconds: 176.67\n",
      "Epoch   7/100 Batch  580/781 - Loss:  1.019, Seconds: 123.86\n",
      "Epoch   7/100 Batch  600/781 - Loss:  1.155, Seconds: 160.57\n",
      "Epoch   7/100 Batch  620/781 - Loss:  1.081, Seconds: 229.01\n",
      "Epoch   7/100 Batch  640/781 - Loss:  1.144, Seconds: 213.84\n",
      "Epoch   7/100 Batch  660/781 - Loss:  1.352, Seconds: 215.01\n",
      "Epoch   7/100 Batch  680/781 - Loss:  1.281, Seconds: 197.15\n",
      "Epoch   7/100 Batch  700/781 - Loss:  1.159, Seconds: 195.18\n",
      "Epoch   7/100 Batch  720/781 - Loss:  1.155, Seconds: 210.13\n",
      "Epoch   7/100 Batch  740/781 - Loss:  1.104, Seconds: 187.91\n",
      "Epoch   7/100 Batch  760/781 - Loss:  1.114, Seconds: 176.62\n",
      "Average loss for this update: 1.159\n",
      "New Record!\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-158-3fdb2639b0d7>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     29\u001b[0m                  \u001b[0msummary_length\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0msummaries_lengths\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     30\u001b[0m                  \u001b[0mtext_length\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mtexts_lengths\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 31\u001b[0;31m                  keep_prob: keep_probability})\n\u001b[0m\u001b[1;32m     32\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     33\u001b[0m             \u001b[0mbatch_loss\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0mloss\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m//anaconda/lib/python3.5/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m    776\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    777\u001b[0m       result = self._run(None, fetches, feed_dict, options_ptr,\n\u001b[0;32m--> 778\u001b[0;31m                          run_metadata_ptr)\n\u001b[0m\u001b[1;32m    779\u001b[0m       \u001b[0;32mif\u001b[0m \u001b[0mrun_metadata\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    780\u001b[0m         \u001b[0mproto_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTF_GetBuffer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrun_metadata_ptr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m//anaconda/lib/python3.5/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run\u001b[0;34m(self, handle, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m    980\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mfinal_fetches\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mfinal_targets\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    981\u001b[0m       results = self._do_run(handle, final_targets, final_fetches,\n\u001b[0;32m--> 982\u001b[0;31m                              feed_dict_string, options, run_metadata)\n\u001b[0m\u001b[1;32m    983\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    984\u001b[0m       \u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m//anaconda/lib/python3.5/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_run\u001b[0;34m(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m   1030\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mhandle\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1031\u001b[0m       return self._do_call(_run_fn, self._session, feed_dict, fetch_list,\n\u001b[0;32m-> 1032\u001b[0;31m                            target_list, options, run_metadata)\n\u001b[0m\u001b[1;32m   1033\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1034\u001b[0m       return self._do_call(_prun_fn, self._session, handle, feed_dict,\n",
      "\u001b[0;32m//anaconda/lib/python3.5/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_call\u001b[0;34m(self, fn, *args)\u001b[0m\n\u001b[1;32m   1037\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0m_do_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1038\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1039\u001b[0;31m       \u001b[0;32mreturn\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1040\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0merrors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mOpError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1041\u001b[0m       \u001b[0mmessage\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcompat\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mas_text\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmessage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m//anaconda/lib/python3.5/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run_fn\u001b[0;34m(session, feed_dict, fetch_list, target_list, options, run_metadata)\u001b[0m\n\u001b[1;32m   1019\u001b[0m         return tf_session.TF_Run(session, options,\n\u001b[1;32m   1020\u001b[0m                                  \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtarget_list\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1021\u001b[0;31m                                  status, run_metadata)\n\u001b[0m\u001b[1;32m   1022\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1023\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_prun_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msession\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "# Train the Model\n",
    "learning_rate_decay = 0.95\n",
    "min_learning_rate = 0.0005\n",
    "display_step = 20 # Check training loss after every 20 batches\n",
    "stop_early = 0 \n",
    "stop = 3 # If the update loss does not decrease in 3 consecutive update checks, stop training\n",
    "per_epoch = 3 # Make 3 update checks per epoch\n",
    "update_check = (len(sorted_texts_short)//batch_size//per_epoch)-1\n",
    "\n",
    "update_loss = 0 \n",
    "batch_loss = 0\n",
    "summary_update_loss = [] # Record the update losses for saving improvements in the model\n",
    "\n",
    "checkpoint = \"best_model.ckpt\" \n",
    "with tf.Session(graph=train_graph) as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    \n",
    "    # If we want to continue training a previous session\n",
    "    #loader = tf.train.import_meta_graph(\"./\" + checkpoint + '.meta')\n",
    "    #loader.restore(sess, checkpoint)\n",
    "    \n",
    "    for epoch_i in range(1, epochs+1):\n",
    "        update_loss = 0\n",
    "        batch_loss = 0\n",
    "        for batch_i, (summaries_batch, texts_batch, summaries_lengths, texts_lengths) in enumerate(\n",
    "                get_batches(sorted_summaries_short, sorted_texts_short, batch_size)):\n",
    "            start_time = time.time()\n",
    "            _, loss = sess.run(\n",
    "                [train_op, cost],\n",
    "                {input_data: texts_batch,\n",
    "                 targets: summaries_batch,\n",
    "                 lr: learning_rate,\n",
    "                 summary_length: summaries_lengths,\n",
    "                 text_length: texts_lengths,\n",
    "                 keep_prob: keep_probability})\n",
    "\n",
    "            batch_loss += loss\n",
    "            update_loss += loss\n",
    "            end_time = time.time()\n",
    "            batch_time = end_time - start_time\n",
    "\n",
    "            if batch_i % display_step == 0 and batch_i > 0:\n",
    "                print('Epoch {:>3}/{} Batch {:>4}/{} - Loss: {:>6.3f}, Seconds: {:>4.2f}'\n",
    "                      .format(epoch_i,\n",
    "                              epochs, \n",
    "                              batch_i, \n",
    "                              len(sorted_texts_short) // batch_size, \n",
    "                              batch_loss / display_step, \n",
    "                              batch_time*display_step))\n",
    "                batch_loss = 0\n",
    "\n",
    "            if batch_i % update_check == 0 and batch_i > 0:\n",
    "                print(\"Average loss for this update:\", round(update_loss/update_check,3))\n",
    "                summary_update_loss.append(update_loss)\n",
    "                \n",
    "                # If the update loss is at a new minimum, save the model\n",
    "                if update_loss <= min(summary_update_loss):\n",
    "                    print('New Record!') \n",
    "                    stop_early = 0\n",
    "                    saver = tf.train.Saver() \n",
    "                    saver.save(sess, checkpoint)\n",
    "\n",
    "                else:\n",
    "                    print(\"No Improvement.\")\n",
    "                    stop_early += 1\n",
    "                    if stop_early == stop:\n",
    "                        break\n",
    "                update_loss = 0\n",
    "            \n",
    "                    \n",
    "        # Reduce learning rate, but not below its minimum value\n",
    "        learning_rate *= learning_rate_decay\n",
    "        if learning_rate < min_learning_rate:\n",
    "            learning_rate = min_learning_rate\n",
    "        \n",
    "        if stop_early == stop:\n",
    "            print(\"Stopping Training.\")\n",
    "            break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Making Our Own Summaries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To see the quality of the summaries that this model can generate, you can either create your own review, or use a review from the dataset. You can set the length of the summary to a fixed value, or use a random value like I have here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def text_to_seq(text):\n",
    "    '''Prepare the text for the model'''\n",
    "    \n",
    "    text = clean_text(text)\n",
    "    return [vocab_to_int.get(word, vocab_to_int['<UNK>']) for word in text.split()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 167,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./best_model.ckpt\n",
      "Original Text: love individual oatmeal cups found years ago sam quit selling sound big lots quit selling found target expensive buy individually trilled get entire case time go anywhere need water microwave spoon know quaker flavor packets\n",
      "\n",
      "Text\n",
      "  Word Ids:    [70595, 18808, 668, 45565, 51927, 51759, 32488, 13510, 32036, 59599, 11693, 444, 23335, 32036, 59599, 51927, 67316, 726, 24842, 50494, 48492, 1062, 44749, 38443, 42344, 67973, 14168, 7759, 5347, 29528, 58763, 18927, 17701, 20232, 47328]\n",
      "  Input Words: love individual oatmeal cups found years ago sam quit selling sound big lots quit selling found target expensive buy individually trilled get entire case time go anywhere need water microwave spoon know quaker flavor packets\n",
      "\n",
      "Summary\n",
      "  Word Ids:       [70595, 28738]\n",
      "  Response Words: love it\n"
     ]
    }
   ],
   "source": [
    "# Create your own review or use one from the dataset\n",
    "#input_sentence = \"I have never eaten an apple before, but this red one was nice. \\\n",
    "                  #I think that I will try a green apple next time.\"\n",
    "#text = text_to_seq(input_sentence)\n",
    "random = np.random.randint(0,len(clean_texts))\n",
    "input_sentence = clean_texts[random]\n",
    "text = text_to_seq(clean_texts[random])\n",
    "\n",
    "checkpoint = \"./best_model.ckpt\"\n",
    "\n",
    "loaded_graph = tf.Graph()\n",
    "with tf.Session(graph=loaded_graph) as sess:\n",
    "    # Load saved model\n",
    "    loader = tf.train.import_meta_graph(checkpoint + '.meta')\n",
    "    loader.restore(sess, checkpoint)\n",
    "\n",
    "    input_data = loaded_graph.get_tensor_by_name('input:0')\n",
    "    logits = loaded_graph.get_tensor_by_name('predictions:0')\n",
    "    text_length = loaded_graph.get_tensor_by_name('text_length:0')\n",
    "    summary_length = loaded_graph.get_tensor_by_name('summary_length:0')\n",
    "    keep_prob = loaded_graph.get_tensor_by_name('keep_prob:0')\n",
    "    \n",
    "    #Multiply by batch_size to match the model's input parameters\n",
    "    answer_logits = sess.run(logits, {input_data: [text]*batch_size, \n",
    "                                      summary_length: [np.random.randint(5,8)], \n",
    "                                      text_length: [len(text)]*batch_size,\n",
    "                                      keep_prob: 1.0})[0] \n",
    "\n",
    "# Remove the padding from the tweet\n",
    "pad = vocab_to_int[\"<PAD>\"] \n",
    "\n",
    "print('Original Text:', input_sentence)\n",
    "\n",
    "print('\\nText')\n",
    "print('  Word Ids:    {}'.format([i for i in text]))\n",
    "print('  Input Words: {}'.format(\" \".join([int_to_vocab[i] for i in text])))\n",
    "\n",
    "print('\\nSummary')\n",
    "print('  Word Ids:       {}'.format([i for i in answer_logits if i != pad]))\n",
    "print('  Response Words: {}'.format(\" \".join([int_to_vocab[i] for i in answer_logits if i != pad])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Examples of reviews and summaries:\n",
    "- Review(1): The coffee tasted great and was at such a good price! I highly recommend this to everyone!\n",
    "- Summary(1): great coffee\n",
    "\n",
    "\n",
    "- Review(2): This is the worst cheese that I have ever bought! I will never buy it again and I hope you won't either!\n",
    "- Summary(2): omg gross gross\n",
    "\n",
    "\n",
    "- Review(3): love individual oatmeal cups found years ago sam quit selling sound big lots quit selling found target expensive buy individually trilled get entire case time go anywhere need water microwave spoon know quaker flavor packets\n",
    "- Summary(3): love it"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I hope that you found this project to be rather interesting and informative. One of my main recommendations for working with this dataset and model is either use a GPU, a subset of the dataset, or plenty of time to train your model. As you might be able to expect, the model will not be able to make good predictions just by seeing many reviews, it needs so see the reviews many times to be able to understand the relationship between words and between descriptions & summaries. \n",
    "\n",
    "In short, I'm pleased with how well this model performs. After creating numerous reviews and checking those from the dataset, I can happily say that most of the generated summaries are appropriate, some of them are great, and some of them make mistakes. I'll try to improve this model and if it gets better, I'll update my GitHub.\n",
    "\n",
    "Thanks for reading!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}