LSTM_tensorflow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Recurrent Neural Networks Poem/ Text Generation\n",
    "by Iason and Paul\n",
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b> Abstract </b><br>\n",
    "In this notebook we work on a project in which we use tensorflow to create a RNN for language modelling. We use a multilayer network with LSTM cells and try different parameters to achieve the best performance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2> 1. Introduction </h2>\n",
    "<br>\n",
    "<h3> The Idea </h3>\n",
    "One of the amazing applications of Recurrent Neural Networks is natural language processing. More specifically, different types of RNNs have been used to generate new text after training on existing one. In this assignment we attempt to train a Recurrent Neural Network on the two epic poems by Ancient Greek poet Homer and create a new one using the vocabulary. \n",
    "<br>\n",
    "\n",
    "<h3> The Plan </h3>\n",
    "The process involves the RNN attempting to predict what letters will follow given certain strings of letters (sentences). A dictionary is created with all the possible characters in the corpus and then they are one hot encoded. During the training phase, batches of phrases with same number of letters are fed to the neural network and in each time step the next letter is attempted to be predicted with the labels always being the next letter after the input letter. In the testing phase, given the start of a sentence, the RNN attempts to sample new letters to create its own text.\n",
    "<br>\n",
    "\n",
    "<h3> The Reality </h3>\n",
    "It turned out to be very hard to create a functioning RNN that actually is able to create words. And yet the structures of 'sentences' our network creates do not really make sense. We worked really hard and read a lot of different blogs, papers and forum posts to get this far and we finally achieved to create a working network.\n",
    "\n",
    "<b> The positive we learned </b><br>\n",
    "This task required a lot of knowledge in neural networks, language processing and tensorflow, where the knowledge in tensorflow is crucial for working on errors. We learned a lot in all of these aspects! <br>\n",
    "In addition to the general things we read a lot about the state of the art models and how things are done at the moment. We also learned more about the underlying math of neural networks, how they can simply be described as functions and how backpropagation works through time. Furthermore, we learned all about the different parameters, which are not as common as e.g. learning rate or number of neurons. We saw what different activations functions will do to a network as well as the importance of choosing the right optimizer and the right loss function.<br>\n",
    "All of this was very interesting and very helpful for understanding the big picture of neural nets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2> 2. Implementation </h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import datetime\n",
    "import random"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3> 2.1 Data Preparation </h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Text length (number of characters):  698120\n",
      "\n",
      "\n",
      "Text:\n",
      "INTRODUCTION\n",
      "\n",
      "Scepticism is as much the result of knowledge, as knowledge is of\n",
      "scepticism. To be content with what we at present know, is, for the\n",
      "most part, to shut our ears against conviction; since, from the very\n",
      "gradual character of our education, we must continually forget, and\n",
      "emancipate ourselves from, knowledge previously acquired; we must set\n",
      "aside old notions and embrace fresh ones; and, as we learn, we must\n",
      "be daily unlearning something which it has cost us no small labour\n",
      "and anxiety to acquire.\n",
      "\n",
      "And this difficulty attaches itself more closely to an age in which\n",
      "progress has gained a strong ascendency over prejudice, and in which\n",
      "persons and things are, day by day, finding their real level, in lieu\n",
      "of their conventional value. The same principles which have swept\n",
      "away traditional abuses, and which are making rapid havoc among the\n",
      "revenues of sinecurists, and stripping the thin, tawdry veil from\n",
      "attractive superstitions, are working as actively in literature as in\n",
      "society.\n"
     ]
    }
   ],
   "source": [
    "text = open(\"homer_odysse.txt\").read()\n",
    "print('Text length (number of characters): ', len(text))\n",
    "print('\\n')\n",
    "print('Text:')\n",
    "print(text[:1000]) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All the characters that are used in out text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of characters: 76\n",
      "['\\n', ' ', '!', '\"', '&', \"'\", '(', ')', ',', '-', '.', '/', '0', '1', '3', '5', '6', '7', '8', '9', ':', ';', '?', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z', '`', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']\n"
     ]
    }
   ],
   "source": [
    "chars = sorted(list(set(text)))\n",
    "char_size = len(chars)\n",
    "print('Number of characters:', char_size)\n",
    "print(chars)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Encoding our text -> characters to numbers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "char2id = dict((c, i) for i, c in enumerate(chars))\n",
    "id2char = dict((i, c) for i, c in enumerate(chars))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data = [char2id[i] for i in text]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "len_per_section = 50\n",
    "skip = 2\n",
    "sections = []\n",
    "next_chars = []"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating training data already put in an easy shape with len_per_section:<br>\n",
    "(labels are always the next character)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for i in range(0, len(data) - len_per_section, skip):\n",
    "    sections.append(data[i: i + len_per_section])\n",
    "    next_chars.append(data[i+1: i+1 + len_per_section])\n",
    "    \n",
    "training_data = np.array(sections)\n",
    "training_labels = np.array(next_chars)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Checking if all the data/ labels are correct:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data/ label shapes:\n",
      "(349035, 50)\n",
      "(349035, 50)\n"
     ]
    }
   ],
   "source": [
    "print('Data/ label shapes:')\n",
    "print(training_data.shape)\n",
    "print(training_labels.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checking if the labels are correct: \n",
      "[50 52 52 64 67 53  8  0 23 63 53  1 72 58 61 61 58 63 56  1 63 50 69 58 64\n",
      " 63 68  1 60 63 54 72  1 69 57 54 58 67  1 61 50 72 55 70 61  1 61 64 67 53]\n",
      "[52 52 64 67 53  8  0 23 63 53  1 72 58 61 61 58 63 56  1 63 50 69 58 64 63\n",
      " 68  1 60 63 54 72  1 69 57 54 58 67  1 61 50 72 55 70 61  1 61 64 67 53 10]\n"
     ]
    }
   ],
   "source": [
    "print('Checking if the labels are correct: ')\n",
    "print(training_data[-1])\n",
    "print(training_labels[-1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>2.2 RNN construction </h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b> 2.2.1 Sample Function </b><br>\n",
    "We use a function to pick <b>one</b> of the most likely next characters. This is to introduce variance to the text, otherwise we get would just keep repeating the most common words (e.g. the, and, was). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def sample(prediction):\n",
    "    ix = np.random.choice(range(char_size), p=prediction.ravel())\n",
    " \n",
    "    return ix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b> 2.2.2 Hyperparameters </b>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Number of hidden neurons:</b><br>\n",
    "This formula provides a good way of gving you boundaries for the number of neurons you use.\n",
    "\n",
    "<b> Nh = Ns / (α ∗ (Ni + No)) </b>\n",
    "\n",
    "Ni = number of input neurons. <br>\n",
    "No = number of output neurons. <br>\n",
    "Ns = number of samples in training data set. <br>\n",
    "α = an arbitrary scaling factor usually 2-10. <br>\n",
    "(Explanation taken from here: https://stats.stackexchange.com/questions/181/how-to-choose-the-number-of-hidden-layers-and-nodes-in-a-feedforward-neural-netw)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It also seems that lots of people use a <b>power of two</b>, because it makes sense computationally. So we decided to see, if when we use powers of two, the alpha is still in the often suggested range of ca. α = 2 to 10. And it works:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Checking if using <b>powers of two</b> still lead to an acceptable alpha: <br><br>\n",
    "<b> α  = Ns / (Nh * (Ni + No)) </b><br><br>\n",
    "Number of neurons:<br>\n",
    "256 -> α = 8.7<br>\n",
    "512 -> α = 4.48<br>\n",
    "1024 -> α = 2.24\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could use all these numbers of neurons in our RNN, as they are all compatible with the formula. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b> More Parameters </b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "batch_size = 512\n",
    "\n",
    "max_steps = 10001\n",
    "\n",
    "save_every = 100\n",
    "\n",
    "n_neurons = 1024\n",
    "\n",
    "learning_rate = 0.01\n",
    "\n",
    "checkpoint_directory = 'lstm_try'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b> Creating a directory </b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "if tf.gfile.Exists(checkpoint_directory):\n",
    "    tf.gfile.DeleteRecursively(checkpoint_directory)\n",
    "tf.gfile.MakeDirs(checkpoint_directory)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b> 2.2.3 Graph construction </b>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><b> Explanations of the RNN architecture:</b> <br><br>\n",
    "<b> Embeddings: </b><br>\n",
    "Use embeddings instead of one-hot-encoding. Embeddings are starting off basically as one-hot-encoded verctors, for each character(e.g. 'a', '.', 'Z', ...) one vector, but the network learns which characters are related to each other and gives those related characters similar vector representations. It works like so:\n",
    "https://www.tensorflow.org/programmers_guide/embedding<br>\n",
    "\n",
    "\n",
    "<b> Loss function: </b><br>\n",
    "Use a loss function that does not compare to embedded labels/ vectorized characters, but to a \"single specific index for the true class\". <br>Like so: https://www.tensorflow.org/api_docs/python/tf/nn/sparse_softmax_cross_entropy_with_logits\n",
    "\n",
    "<b> Gradient Clipping: </b><br>\n",
    "During the training of many different types of RNNs we have seen that sometimes after a random epoch the loss will suddenly start to go up again, this is due to gradients becoming too large. Thus we want to do gradient clipping to avoid that a single 'bad' batch messing up all our weights. <br>\n",
    "It is done like so: https://www.tensorflow.org/versions/master/api_docs/python/tf/train/Optimizer#processing_gradients_before_applying_them<br>\n",
    "With the help of a function (ClipIfNotNone()) that is needed because tf.clip_by_value breaks if the gradients are None. Function is taken from: https://stackoverflow.com/questions/39295136/gradient-clipping-appears-to-choke-on-none "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /Users/paulschuetze/anaconda/lib/python3.6/site-packages/tensorflow/python/util/tf_should_use.py:170: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.\n",
      "Instructions for updating:\n",
      "Use `tf.global_variables_initializer` instead.\n"
     ]
    }
   ],
   "source": [
    "tf.reset_default_graph()\n",
    "\n",
    "data = tf.placeholder(tf.int32, [None, None], name='data')\n",
    "labels = tf.placeholder(tf.int32, [None, len_per_section], name='labels')\n",
    "\n",
    "# Placeholde to set the init_state size -> is needed to allow us to pass only single characters\n",
    "# to the network after training on sequences\n",
    "b_size = tf.placeholder_with_default(input=batch_size, shape=())\n",
    "\n",
    "# Embeddings\n",
    "char_embeddings = tf.get_variable('char_embeddings', [char_size, n_neurons])\n",
    "embedded_char_ids = tf.nn.embedding_lookup(char_embeddings, data)\n",
    "\n",
    "\n",
    "# LSTM cells\n",
    "cell = tf.nn.rnn_cell.LSTMCell(n_neurons, state_is_tuple=True)\n",
    "# Wrapps around the lstm cells so we can feed it into dynamic_rnn\n",
    "cell = tf.nn.rnn_cell.MultiRNNCell([cell] * 3, state_is_tuple=True)\n",
    "\n",
    "init_state = cell.zero_state(b_size, tf.float32)\n",
    "outputs, final_state = tf.nn.dynamic_rnn(cell, embedded_char_ids, initial_state=init_state)\n",
    "\n",
    "w = tf.Variable(tf.truncated_normal([n_neurons, char_size], -0.1, 0.1))\n",
    "b = tf.Variable(tf.zeros([char_size]))\n",
    " \n",
    "# Modifiy output so we can do matmul and use it in our loss function\n",
    "outputs_reshaped = tf.reshape(outputs, [-1, n_neurons])\n",
    "labels_reshaped = tf.reshape(labels, [-1])\n",
    "\n",
    "# Basically a fully connected layer transformation on our outputs\n",
    "logits = tf.matmul(outputs_reshaped, w) + b\n",
    "\n",
    "\n",
    "loss = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(logits=logits, labels=labels_reshaped))\n",
    "\n",
    "optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)\n",
    "\n",
    "# Gradient clipping\n",
    "gvs = optimizer.compute_gradients(loss)\n",
    "\n",
    "def ClipIfNotNone(grad):\n",
    "    if grad is None:\n",
    "            return grad\n",
    "    return tf.clip_by_value(grad, -1, 1)\n",
    "\n",
    "capped_gvs = [(ClipIfNotNone(grad), var) for grad, var in gvs]\n",
    "training_op = optimizer.apply_gradients(capped_gvs)\n",
    "\n",
    "test_prediction = tf.nn.softmax(logits)\n",
    "\n",
    "init = tf.initialize_all_variables()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b> 2.2.4 Graph execution </b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## with tf.Session() as sess:\n",
    "    init.run()\n",
    "    saver = tf.train.Saver()\n",
    "    offset = 0\n",
    "    \n",
    "    \n",
    "    # Repeat for max_steps\n",
    "    for step in range(max_steps):\n",
    "        \n",
    "        # Walk through the data:\n",
    "        offset = offset % len(training_data)\n",
    "        \n",
    "        # Walking normally, not at the end\n",
    "        if offset <= (len(training_data) - batch_size):\n",
    "            \n",
    "            batch_data = training_data[offset: offset + batch_size]\n",
    "            batch_labels = training_labels[offset: offset + batch_size]\n",
    "            # Update offset, so we use different instances next time:\n",
    "            offset += batch_size\n",
    "            \n",
    "        # In the case we reach the end of our data:\n",
    "        else:\n",
    "            # Add the missing part to the batch, take it from the front of the data:\n",
    "            to_add = batch_size - (len(training_data) - offset)\n",
    "            batch_data = np.concatenate((training_data[offset: len(training_data)], training_data[0: to_add]))\n",
    "            batch_labels = np.concatenate((training_labels[offset: len(training_data)], training_labels[0: to_add]))\n",
    "            offset = to_add\n",
    "        \n",
    "        # Run our model on the data\n",
    "        _, training_loss = sess.run([training_op, loss], feed_dict={data: batch_data, \n",
    "                                                                    labels: batch_labels})\n",
    "        \n",
    "        if step % 10 == 0:\n",
    "            print('training loss at step %d: %.2f (%s)' % (step, training_loss, datetime.datetime.now()))\n",
    "\n",
    "        if step % save_every == 0:\n",
    "            saver.save(sess, checkpoint_directory + '/model', global_step=step)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>2.3 RNN testing </h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we load the trained model and let it create some text. We feed it character by character and let it generate the next letter. This is why we need to have batch_size as a placeholder: it gives the network the variability to accepts single character input (It changes the shape of init_state, so we can pass a single character into our LSTM construction.)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from lstm_try/model-800\n",
      "I. Where as: \"Was my gree,\n",
      "Who let the dreased's blance with the seirs I srines,\n",
      "And I divent Proglid to The seas mall Ghaan adest, the on arm-astrops her on shore tastess thoum thee Itister-potent him\n"
     ]
    }
   ],
   "source": [
    "# Start of with seed character:\n",
    "test_start = 'I'\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    # Load Model: \n",
    "    tf.global_variables_initializer().run()\n",
    "    model = tf.train.latest_checkpoint(checkpoint_directory)\n",
    "    saver = tf.train.Saver()\n",
    "    saver.restore(sess, model)\n",
    "    \n",
    "    # State is none so we are able to accept single character input, could be any another variable as well:\n",
    "    state = None\n",
    "    \n",
    "    # Our text we add to:\n",
    "    test_generated = test_start\n",
    "    current_char = char2id[test_start]\n",
    "    \n",
    "    for i in range(200):\n",
    "        \n",
    "        # Feed data character by character:\n",
    "        if state is not None:\n",
    "             # Get latest character as new input\n",
    "            feed_dict={data: [[current_char]], init_state: state}\n",
    "        else:\n",
    "             # Get latest character as new input\n",
    "            feed_dict={data: [[current_char]], b_size: 1}\n",
    "\n",
    "        preds, state = sess.run([test_prediction , final_state], feed_dict)\n",
    "        # Use sample function\n",
    "        current_char = sample(preds[0]) \n",
    "        \n",
    "        # Add characters to generated text:\n",
    "        test_generated += id2char[current_char]\n",
    "        \n",
    "\n",
    "    print(test_generated)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b> 2.3.1 Results of testing </b><br>\n",
    "We gave the network a seed letter ('I') and let it generate 200 characters (any number would work) after that, the new character is always the new input. We use the sample function to achieve variation in the text. <br>\n",
    "We also did not use a validation error during training or testing, the validation error is basically represented by us judging the created text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " Tests of the model <b>STILL</b> on the <b>TWAIN</b> data: <br><br>\n",
    "<b> 256 neurons, 3 layers, 4700 epochs -> 9 hours of training, averge loss = 1.8</b><br>\n",
    "INFO:tensorflow:Restoring parameters from lstm_try/model-4700<br>\n",
    "I couldn’t coath to let up at street fro invight I didn’t what makes you’ve a dozening dewful jeckon Dogd all over him to place and gay two much one back to get down it.  Yes, or paw uf Sundeading “I’d\n",
    "\n",
    "<b> 512 neurons, 3 layers, 1800 epochs -> 3 hours of training, averge loss = 2.2</b><br>\n",
    "I.Lse.  Whe the herk_lased hord my nasf om out abr this boff boff, of me’s and I want gack shy.  Yared out bods-as lol he and stugtady cackramisck prige, and at on that shixg a shidns\n",
    "girger you beft s\n",
    "\n",
    "<b> 1024 neurons, 3 layers, 950 epochs -> 7 hours of training, averge loss = 1.5</b><br>\n",
    "INFO:tensorflow:Restoring parameters from lstm_try/model-900<br>\n",
    "I wanted down his vance.  They stristered in people them watchers the dogs was\n",
    "all six all swarsing minute up as you people EndOrerackest the more- I’d along these somebody\n",
    "king around he give like the"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b> HOMER data: </b><br>\n",
    "\n",
    "<b> 1024 neurons, 3 layers, 800 epochs -> 10 hours of training, averge loss = 1.9</b><br>\n",
    "INFO:tensorflow:Restoring parameters from lstm_try/model-800<br>\n",
    "I. Where as: \"Was my gree,\n",
    "Who let the dreased's blance with the seirs I srines,\n",
    "And I divent Proglid to The seas mall Ghaan adest, the on arm-astrops her on shore tastess thoum thee Itister-potent him"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br><b>2.3.2 Changing Parameter Results </b><br>\n",
    "\n",
    "<b> Learning Rate </b><br>\n",
    "Does affect how the loss converges and what value it converges to. Too small: loss does not go down quickly, sometimes not at all. Too large: loss goes down but then stops going down further. It turns out that the learning rate is connected to the number of neurons, the bigger the net the smaller the learning rate.\n",
    "<br>\n",
    "\n",
    "<b> Number of Hidden Neurons</b><br>\n",
    "The more neurons we have the faster (in steps) the loss converges and the lower the loss gets, but also the time per step takes longer much longer. \n",
    "<br>\n",
    "\n",
    "<b> Other Parameters </b><br>\n",
    "There is also other things one can change, e.g. the batch_size and sequence length (over how many timesteps we unroll the network). Especially changing the sequence lengths would have a big effect on our results. Making it longer would probably lead to better results in creating text that makes sense. However, this would take much longer training time as well, this is why we settled with a sequence length of 50, a good middle ground."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2> 3. Conclusion </h2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(Comment: For a inbetween steps and a first conclusion refer to the homework#14 assignment, some of the difficulties are shown there.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After it proved difficult to use simple Recurrent Neural Nets to generate text, we decided to switch to an LSTM model in order to achieve better results. First, we started using a single LSTM cell, but it did not seem to do better than the simple RNN. After doing some research we decided to use a more complex model that consisted of three LSTM cells and a single multi-RNN wrapper class. It should be noted that LSTMs prevent gradient vanishing but they do not help against exploding gradients. Therefore, we had to utilize gradient clipping to prevent bad batches that explode the gradients from ruining training.\n",
    "\n",
    "We first trained on text authored by Mark Twain in which the vocabulary consisted of more commonly spoken words than Homer’s does. We thought the quality of the data might be better in a Mark Twain text. We tweaked the different parameters to achieve the best possible results. We found out that using about 0.01 for learning rate, and 1024 neurons is optimal. The network reached and stabilized at an error of about 1.5 in 400 steps and was able to reproduce full words and sometimes, sentences. <br>\n",
    "This is also our final network, which we show above. We decided not to show all our inbetween networks and tries, to make the notebook more comprehensible. But there have been many different networks we ran before achieving the results above. We also believe that with a little bit of parameter tweeking and more training our network is able to get better and better, the core structure seems to support very good performance.<br>\n",
    "For now we just trained our final model on the Homer text Odyssey, which turned out to take longer than on the Twain text (we have no idea why that is, it might be because of the text quality, or it could just be that our computer had a 'bad day'). Also the results are not as good as before, because of the type of language used in the Osyssey."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3> 3.1 Additional improvement ideas </h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the future, there are a few more things that could be changed in order to maximize performance even more. These include adding more LSTM cells, using dropout in non-recurrent layers, or even using more neurons per layer. We could also modify the data in a way to make it simpler. Take out line breaks and unnuasal parts like headings, footers etc., or restrict the vocabulary by taking out unusual characters and replacing them with a dummie variable. <br>\n",
    "These are all ideas that might eventually lead to better training and to a model that might create meaningful text. However, some of them would be too computationally expensive to try without a powerful machine, others just take more time to implement and we do not have all the time to try them all."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sources\n",
    "- https://r2rt.com/recurrent-neural-networks-in-tensorflow-i.html\n",
    "- http://colah.github.io/posts/2014-07-NLP-RNNs-Representations/\n",
    "- https://www.youtube.com/watch?v=ZGU5kIG7b2I\n",
    "- https://www.youtube.com/watch?v=BwmddtPFWtA"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}