tutorial/01_RNN_generator_playground.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# An RNN model to generate sequences\n",
    "RNN models can generate long sequences based on past data. This can be used to predict stock markets, temperatures, traffic or sales data based on past patterns. They can also be adapted to [generate text](https://docs.google.com/presentation/d/18MiZndRCOxB7g-TcCl2EZOElS5udVaCuxnGznLnmOlE/pub?slide=id.g139650d17f_0_1185). The quality of the prediction will depend on training data, network architecture, hyperparameters, the distance in time at which you are predicting and so on. But most importantly, it will depend on wether your training data contains examples of the behaviour you are trying to predict.\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "<ol>\n",
    "Things to do:<br/>\n",
    "<li> [Choose a waveform](#assignment1)\n",
    "<li> Execute the entire notebook. See results at the bottom: not great...\n",
    "<li> [Implement the RNN model](#assignment2) and try again\n",
    "<li> Check that your state is passed around correctly:\n",
    "    <ol>\n",
    "    <li> Did you use `dynamic_rnn(initial_state=Hin)` [in your model](#model) ?\n",
    "    <li> [During inference](#assignment3A): check the state (hint: it's OK)\n",
    "    <li> [In the training loop](#assignment3): check the state (hint: it's a bug)\n",
    "    <li> [When batching your data](#assignment3): check the state (hint: it's a bug - [see this slide](https://docs.google.com/presentation/d/18MiZndRCOxB7g-TcCl2EZOElS5udVaCuxnGznLnmOlE/pub?slide=id.g139650d17f_0_584) to understand why then use `rnn_minibatch_sequencer` instead of `dumb_minibatch_sequencer`)\n",
    "    </ol>\n",
    "<li> [Make the predictions more robust](#assignment4). You can try any of the following:\n",
    "    <ol>\n",
    "    <li> Use GRUCell instead of BasicRNNCell [in your model](#model)\n",
    "    <li> Train longer NB_EPOCHS 5 -> 10 [in your training loop](#training)\n",
    "    <li> Larger SEQLEN (16->32) [in hyperparameters](#hyperparameters)\n",
    "    <li> Use a stacked RNN cell with 2 layers with `tf.nn.rnn_cell.MultiRNNCell` [in your model](#model)\n",
    "    <li> Use dropout between the RNN cell layers [in your model](#model)\n",
    "    </ol>\n",
    "</ol>\n",
    "    \n",
    "Play with these options until you get a good fit for at least 128 predicted samples. You can then try a [different waveform](#assignment1).\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "import numpy as np\n",
    "from matplotlib import pyplot as plt\n",
    "import utils_prettystyle\n",
    "import utils_batching\n",
    "import tensorflow as tf\n",
    "import math\n",
    "print(\"Tensorflow version: \" + tf.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"assignment1\"></a>\n",
    "## Generate fake dataset\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "**Assignment #1**: Choose a waveform. Three possible choices on the next line: 0, 1 or 2\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "WAVEFORM_SELECT = 0 # select 0, 1 or 2\n",
    "\n",
    "def create_time_series(datalen):\n",
    "    # good waveforms\n",
    "    frequencies = [(0.2, 0.15), (0.35, 0.3), (0.6, 0.55)]\n",
    "    freq1, freq2 = frequencies[WAVEFORM_SELECT]\n",
    "    noise = [np.random.random()*0.2 for i in range(datalen)]\n",
    "    x1 = np.sin(np.arange(0,datalen) * freq1)  + noise\n",
    "    x2 = np.sin(np.arange(0,datalen) * freq2)  + noise\n",
    "    x = x1 + x2\n",
    "    return x.astype(np.float32)\n",
    "\n",
    "DATA_SEQ_LEN = 1024*128\n",
    "data = create_time_series(DATA_SEQ_LEN)\n",
    "plt.plot(data[:512])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"hyperparameters\"></a>\n",
    "## Hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "RNN_CELLSIZE = 64   # size of the RNN cells\n",
    "NLAYERS = 1         # number of stacked RNN cells (needed for tensor shapes but code must be changed manually)\n",
    "SEQLEN = 16         # unrolled sequence length\n",
    "BATCHSIZE = 32      # mini-batch size\n",
    "DROPOUT_PKEEP = 0.7 # probability of neurons not being dropped (should be between 0.5 and 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize training sequences\n",
    "This is what the neural network will see during training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The function dumb_minibatch_sequencer splits the data into batches of sequences sequentially.\n",
    "for samples, labels, epoch in utils_batching.dumb_minibatch_sequencer(data, BATCHSIZE, SEQLEN, nb_epochs=1):\n",
    "    break\n",
    "print(\"Sample shape: \" + str(samples.shape))\n",
    "print(\"Label shape: \" + str(labels.shape))\n",
    "print(\"Excerpt from first batch:\")\n",
    "subplot = 231\n",
    "for i in range(6):\n",
    "    plt.subplot(subplot)\n",
    "    plt.plot(samples[i])\n",
    "    subplot += 1\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"assignment2\"></a>\n",
    "<a name=\"model\"></a>\n",
    "## The model definition\n",
    "When executed, this function instantiates the Tensorflow graph for our model.\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "**Assignment #2**: implement a single-layer RNN based on a basic RNN cell (tf.nn.rnn_cell.BasicRNNCell)\n",
    "</div>\n",
    "\n",
    "![deep RNN schematic](images/deep_rnn.png)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Really dumb model...\n",
    "def model_fn(features, Hin, labels, dropout_pkeep):\n",
    "    # inputs shapes during training (for inference, we will use BATCHSIZE=1 and SEQLEN=1):\n",
    "    # features [BATCHSIZE, SEQLEN, 1]\n",
    "    # labels [BATCHSIZE, SEQLEN, 1]\n",
    "    # Hin [BATCHSIZE, RNN_CELLSIZE*NLAYERS]\n",
    "    X = features\n",
    "    batchsize = tf.shape(X)[0] # determined dynamically\n",
    "    seqlen = tf.shape(X)[1] # determined dynamically\n",
    "    \n",
    "    # Goals:\n",
    "    # Tranform input \"X=features\" into output \"Yout\"\n",
    "    # Tranform input \"Hin\" into output \"H\" (these will be input and output states in an RNN)\n",
    "    # Compute a loss between \"Yout\" and \"labels\" and minimize it\n",
    "    \n",
    "    # dummy model that does almost nothing (one trainable variable is needed)\n",
    "    Yr = X * tf.Variable(tf.ones([]))\n",
    "    H = Hin\n",
    "    \n",
    "    # TODO: create a tf.nn.rnn_cell.GRUCell\n",
    "    # TODO: unroll the cell using tf.nn.dynamic_rnn\n",
    "    # TODO: add a regression head using tf.layers.dense with just 1 neuron and no activation\n",
    "    # TIP: you might need to reshape the sequence of outputs from the unrolled RNN (tf.reshape)\n",
    "    \n",
    "    # Yr[BATCHSIZE, SEQLEN, 1]\n",
    "    Yout = Yr[:,-1,:]\n",
    "    # Last output in sequence Yout [BATCHSIZE, 1]\n",
    "    \n",
    "    # shapes:\n",
    "    # Yr [BATCHSIZE, SEQLEN, 1]\n",
    "    # labels[BATCHSIZE, SEQLEN, 1]\n",
    "    loss = tf.losses.mean_squared_error(Yr, labels)\n",
    "    optimizer = tf.train.AdamOptimizer(learning_rate=0.01)\n",
    "    train_op = optimizer.minimize(loss)\n",
    "    \n",
    "    # output shapes:\n",
    "    # Yout [BATCHSIZE, 1]\n",
    "    # H [BATCHSIZE, RNN_CELLSIZE*NLAYERS]\n",
    "    return Yout, H, loss, train_op"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Instantiate the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# placeholder for inputs\n",
    "Hin = tf.placeholder(tf.float32, [None, RNN_CELLSIZE * NLAYERS])\n",
    "samples = tf.placeholder(tf.float32, [None, None, 1]) # [BATCHSIZE, SEQLEN, 1]\n",
    "labels = tf.placeholder(tf.float32, [None, None, 1]) # [BATCHSIZE, SEQLEN, 1]\n",
    "dropout_pkeep = tf.placeholder(tf.float32)\n",
    "\n",
    "# instantiate the model\n",
    "Yout, H, loss, train_op = model_fn(samples, Hin, labels, dropout_pkeep)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"assignment3A\"></a>\n",
    "<a name=\"inference\"></a>\n",
    "## Inference\n",
    "This is a generative model: run one trained RNN cell in a loop\n",
    "\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "**Assignment #3.A**: Check that the RNN state is passed around correctly (hint: it's OK)\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def prediction_run(prime_data, run_length):\n",
    "    H_ = np.zeros([1, RNN_CELLSIZE * NLAYERS]) # zero state initially\n",
    "    Yout_ = np.zeros([1, 1])\n",
    "    data_len = prime_data.shape[0]\n",
    "\n",
    "    # prime the state from data\n",
    "    if data_len > 0:\n",
    "        Yin = np.array(prime_data)\n",
    "        Yin = np.reshape(Yin, [1, data_len, 1]) # reshape as one sequence\n",
    "        feed = {Hin: H_, samples: Yin, dropout_pkeep: 1.0} # no dropout during inference\n",
    "        Yout_, H_ = sess.run([Yout, H], feed_dict=feed)\n",
    "    \n",
    "    # run prediction\n",
    "    # To generate a sequence, run a trained cell in a loop passing as input and input state\n",
    "    # respectively the output and output state from the previous iteration.\n",
    "    results = []\n",
    "    for i in range(run_length):\n",
    "        Yout_ = np.reshape(Yout_, [1, 1, 1]) # batch of a single sequence of a single vector with one element\n",
    "        feed = {Hin: H_, samples: Yout_, dropout_pkeep: 1.0} # no dropout during inference\n",
    "        Yout_, H_ = sess.run([Yout, H], feed_dict=feed)\n",
    "        results.append(Yout_[0,0])\n",
    "        \n",
    "    return np.array(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialize Tensorflow session\n",
    "This resets all neuron weights and biases to initial random values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# first input state\n",
    "Hzero = np.zeros([BATCHSIZE, RNN_CELLSIZE * NLAYERS])\n",
    "# variable initialization\n",
    "sess = tf.Session()\n",
    "init = tf.global_variables_initializer()\n",
    "sess.run([init])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"assignment3\"></a>\n",
    "<a name=\"training\"></a>\n",
    "## The training loop\n",
    "You can re-execute this cell to continue training\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "**Assignment #3**: find and resolve RNN state bugs.<br?>\n",
    "**hint**: there are 2 bugs. One in the core of the training loop and one in the way the data was sliced into batches of sequences. Special care is needed when batching sequences for an RNN. [See this slide](https://docs.google.com/presentation/d/18MiZndRCOxB7g-TcCl2EZOElS5udVaCuxnGznLnmOlE/pub?slide=id.g139650d17f_0_584) to understand the situation. You can fix it by using `rnn_minibatch_sequencer` instead of `dumb_minibatch_sequencer`)\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "NB_EPOCHS = 5\n",
    "\n",
    "H_ = Hzero\n",
    "losses = []\n",
    "indices = []\n",
    "for i, (next_samples, next_labels, epoch) in enumerate(utils_batching.dumb_minibatch_sequencer(data, BATCHSIZE, SEQLEN, nb_epochs=NB_EPOCHS)):\n",
    "    next_samples = np.expand_dims(next_samples, axis=2) # model wants 3D inputs [BATCHSIZE, SEQLEN, 1] \n",
    "    next_labels = np.expand_dims(next_labels, axis=2)\n",
    "\n",
    "    feed = {Hin: Hzero, samples: next_samples, labels: next_labels, dropout_pkeep: DROPOUT_PKEEP}\n",
    "    Yout_, _, loss_, _ = sess.run([Yout, H, loss, train_op], feed_dict=feed)\n",
    "    # print progress\n",
    "    if i%100 == 0:\n",
    "        print(\"epoch \" + str(epoch) + \", batch \" + str(i) + \", loss=\" + str(np.mean(loss_)))\n",
    "    if i%10 == 0:\n",
    "        losses.append(np.mean(loss_))\n",
    "        indices.append(i)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.ylim(ymax=np.amax(losses[1:])) # ignore first value for scaling\n",
    "plt.plot(indices, losses)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PRIMELEN=256\n",
    "RUNLEN=512\n",
    "OFFSET=20\n",
    "\n",
    "prime_data = data[OFFSET:OFFSET+PRIMELEN]\n",
    "\n",
    "results = prediction_run(prime_data, RUNLEN)\n",
    "\n",
    "disp_data = data[OFFSET:OFFSET+PRIMELEN+RUNLEN]\n",
    "colors = plt.rcParams['axes.prop_cycle'].by_key()['color']\n",
    "plt.subplot(211)\n",
    "plt.text(PRIMELEN,2.5,\"DATA |\", color=colors[1], horizontalalignment=\"right\")\n",
    "plt.text(PRIMELEN,2.5,\"| PREDICTED\", color=colors[0], horizontalalignment=\"left\")\n",
    "displayresults = np.ma.array(np.concatenate((np.zeros([PRIMELEN]), results)))\n",
    "displayresults = np.ma.masked_where(displayresults == 0, displayresults)\n",
    "plt.plot(displayresults)\n",
    "displaydata = np.ma.array(np.concatenate((prime_data, np.zeros([RUNLEN]))))\n",
    "displaydata = np.ma.masked_where(displaydata == 0, displaydata)\n",
    "plt.plot(displaydata)\n",
    "plt.subplot(212)\n",
    "plt.text(PRIMELEN,2.5,\"DATA |\", color=colors[1], horizontalalignment=\"right\")\n",
    "plt.text(PRIMELEN,2.5,\"| +PREDICTED\", color=colors[0], horizontalalignment=\"left\")\n",
    "plt.plot(displayresults)\n",
    "plt.plot(disp_data)\n",
    "RMSELEN=128\n",
    "plt.axvspan(PRIMELEN, PRIMELEN+RMSELEN, color='grey', alpha=0.1, ymin=0.05, ymax=0.95)\n",
    "plt.show()\n",
    "\n",
    "rmse = math.sqrt(np.mean((data[OFFSET+PRIMELEN:OFFSET+PRIMELEN+RMSELEN] - results[:RMSELEN])**2))\n",
    "print(\"RMSE on {} predictions (shaded area): {}\".format(RMSELEN, rmse))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"assignment4\"></a>\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "**Assignment #4**: Make the predictions more robust. You can try any of the following:\n",
    "<ol>\n",
    "    <li> Use GRUCell instead of BasicRNNCell [in your model](#model)</li>\n",
    "    <li> Train longer NB_EPOCHS 5 -> 10 [in your training loop](#training)</li>\n",
    "    <li> Larger SEQLEN (16->32) [in hyperparameters](#hyperparameters)</li>\n",
    "    <li> Use a stacked RNN cell with 2 layers with `tf.nn.rnn_cell.MultiRNNCell` [in your model](#model)</li>\n",
    "    <li> Use dropout between the RNN cell layers [in your model](#model)</li>\n",
    "</ol>\n",
    "Play with these options until you get a good fit for at least 128 predicted samples. You can then try a [different waveform](#assignment1).\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "you may not use this file except in compliance with the License.\n",
    "You may obtain a copy of the License at\n",
    "[http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0)\n",
    "Unless required by applicable law or agreed to in writing, software\n",
    "distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "See the License for the specific language governing permissions and\n",
    "limitations under the License."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}