practical4.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Practical 4: Solution RELEASE",
      "version": "0.3.2",
      "views": {},
      "default_view": {},
      "provenance": [
        {
          "file_id": "0B6frZAN2FTPCa2VFZzBFVFBBWDA",
          "timestamp": 1503668999486
        }
      ]
    }
  },
  "cells": [
    {
      "metadata": {
        "id": "-2pzsVf_4e6E",
        "colab_type": "text"
      },
      "source": [
        "# DL Indaba Practical 4\n",
        "# Gated Recurrent Models (GRUs and LSTMs)\n",
        "\n",
        "**Introduction**\n",
        "\n",
        "So far we have looked at feedforward models which learn to map a single input **x** to a label (prediction) **y**. However, a lot of real-world data comes in the form of sequences, for example the words in natural languages, phonemes in speech, and so forth. In this practical we move from feed-forward models to ***sequence models*** which are designed to specifically model the **dependencies** between inputs in sequences of data that change over time. \n",
        "\n",
        "We will again apply this model to predict the labels of the handwritten MNIST images (yes! images can be thought of as **sequences of pixels**!). Then we will show how a very small change turns our MNIST classifier into a **language model** that learns the distribution of words in language!\n",
        "\n",
        "**What is expected of you:**\n",
        " * **TODO: Stipulate desired outcomes (with time-limits).**"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {
        "id": "YZTO9IIiWQNR",
        "colab_type": "text"
      },
      "source": [
        "# Recurrent Neural Networks (RNNs)\n",
        "\n",
        "**NOTE**: You can safely skip the first section below if you know RNNs. Skim the second and dip into the third, but be sure to resurface back at \"Putting it all together\"!\n",
        "\n",
        "## The intuition\n",
        "\n",
        "RNNs generalize feedforward networks (FFNs) to be able to work with sequential data. FFNs take an input (e.g. an image) and immediately produce an output (e.g. a digit class).  RNNs, on the other hand, consider the data sequentially and remembers what it has seen in the past in order to make new predictions about the future observations.\n",
        "\n",
        "To understand this distinction, consider the example where we want to label words as the part-of-speech categories that they belong to: E.g. for the input sentence 'I want a duck' and 'He had to duck', we want our model to predict that duck is a noun in the first sentence and a verb in the second. To do this successfully, the model needs to be aware of the surrounding context.  However, if we feed a FFN model only one word at a time, how could it know the difference? If we want to feed it all the words at once, how do we deal with the fact that sentences are of different lengths? (We *could* try to find a trade-off by feeding it *windows of words*...)\n",
        "\n",
        "RNNs solve this issue by processing the sentence word-by-word, and maintaining an internal state summarizing what it has seen so far. This applies not only to words, but also to phonemes in speech, or even, as we will see, pixels of an image.\n",
        "\n",
        "## The RNN API\n",
        "\n",
        "Feedforward neural networks operate on vectors of fixed size. As we have done before, we could think of the \"API\" of feedforward models as follows:\n"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {
        "id": "zBV_CSuvWPm5",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "class FeedForwardModel():\n",
        "  \n",
        "  # ...\n",
        "  \n",
        "  def forward(self, x):\n",
        "    '''One example of a '''\n",
        "    # Compute activations on the hidden layer.\n",
        "    hidden_layer = np.tanh(np.dot(self.W_xh, x))\n",
        "      \n",
        "    # Compute the (linear) output layer activations.     \n",
        "    output = np.dot(self.W_ho, hidden_layer)\n",
        "    \n",
        "    return output"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "oU95UPGzWXOR",
        "colab_type": "text"
      },
      "source": [
        "Recurrent neural networks (RNNs) generalize this idea to operating on **sequences of vectors**. To process sequences, the model has an internal **state** which gets updated with each new observation. Computationally, one can think of this as recursively applying a function `recurrent_fn` to update the state of the model based on each new input in the sequence:"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {
        "id": "FiNLIPNhWbiZ",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "class RecurrentModel():\n",
        "  \n",
        "  # ...\n",
        "\n",
        "  def recurrent_fn(self, x, prev_state):\n",
        "    '''Process the current input and the previous state and produce an output and a new state.'''\n",
        "    # Compute the new state based on the previous state and current input.  \n",
        "    new_state = np.tanh(np.dot(self.W_hh, prev_state) + np.dot(self.W_xh, x))\n",
        "\n",
        "    # Compute the output vector.\n",
        "    y = np.dot(self.W_hy, new_state)\n",
        "\n",
        "    return new_state, y\n",
        "\n",
        "\n",
        "  def forward(self, data_sequence):\n",
        "\n",
        "    state = self.init_state()\n",
        "    all_states = [state]\n",
        "    last_output = None\n",
        "\n",
        "    for x in data_sequence:\n",
        "\n",
        "      new_state, last_output = recurrent_fn(x, state)\n",
        "      all_states.append(new_state)\n",
        "      state = new_state\n",
        "\n",
        "    return all_states, last_output"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "w-DG9N5tEdaD",
        "colab_type": "text"
      },
      "source": [
        "## Putting this together\n",
        "\n",
        "Look at the definition of `hidden_layer` in `FeedForwardModel.forward()` and `new_state` in `RecurrentModel.forward()`. If you're more comfortable with math, compare the expression for computing the hidden layer of a feedforward neural network:\n",
        "\n",
        "*  $h = \\sigma(W_{xh}x)$\n",
        "\n",
        "to the expression for computing the hidden layer at time step $t$ in an RNN:\n",
        "  \n",
        "* $h_t = \\sigma(W_{hh}h_{t-1} + W_{xh}x_t)$\n",
        "  \n",
        "**QUESTIONS**:\n",
        " * How are they similar? \n",
        " * How are they different?\n",
        " * Why is $W_{hh}$ called \"recurrent\" weights?\n",
        " \n",
        "**NOTE**: Spend a few min to think about/discuss this before you move on. \n",
        "\n",
        "\n",
        "## 'Unrolling' the network\n",
        "\n",
        "Imagine we are trying to classify sequences `X` into labels `y` (for now, let's keep it abstract). After running the `forward()` function of our RNN defined above on `X`, we would have a list of internal states of the model at each sequence position, and the final state of the network. This process is called **unrolling in time**, because you can think of it as unrolling the *computation graph* defined by the RNN `forward` function, over the inputs at each position of the sequence.  RNNs are often used to model **time series data**, and therefore these positions are referred to as **time-steps**, hence, \"unrolling over time\".\n",
        "\n",
        "**TODO(sgouws)**: include graph to display this, e.g. https://r2rt.com/static/images/NH_StateLoop.png\n",
        "\n",
        "> **We can therefore think of an RNN as a composition of identical feedforward neural networks (with replicated/tied weights), one for each moment or step in time. **\n",
        "\n",
        "These feedforward functions (i.e. our `recurrent_fn` above) are typically referred to as **cells**, and the only restriction on its API is that the cell function needs to be a differentiable function that can map an input and a state vector to an output and a new state vector. What we have shown above is called the **vanilla RNN**, but there are many more possibilities. In this practical, we will build up to a family of **gated-recurrent cells**. One of the most popular variants is called the **Long short-term memory** cell. But we're getting ahead of ourselves."
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {
        "id": "kLweKkB7F4j4",
        "colab_type": "text"
      },
      "source": [
        "## Training RNNs: (Truncated) Back-prop through Time\n",
        "\n",
        "RNNs model sequential data, and are designed to capture how ***outputs*** at the current time step are influenced by the ***inputs*** that came before them. This is referred to as **long-range dependencies**. At a high level, this allows the model to `remember` what it has seen so far in order to better contextualize what it is seeing at the moment (think about how knowing the context of the sentence or conversation can sometimes help one to better figure out the intended meaning of a misheard word or ambiguous statement). It is what makes these models so powerful, but it is also what makes them so hard to train!\n",
        "\n",
        "### BPTT: A quick theoretical overview\n",
        "\n",
        "The most well-known algorithm for training RNNs is called **back-propagation through time (BPTT)** (there are other algorithms). BPTT conceptually amounts to unrolling the computations of the RNN over time, computing the errors, and backpropagating the gradients through the unrolled graph structure.  Ideally we want to unroll the graph up to the maximum sequence length, however in practice, since sequence lengths vary and memory is limited, we only end up unrolling sequences up to some length $T$. This is called **truncated BPTT**, and is the most used variant of BPTT.\n",
        "\n",
        "At a high level, there are two main issues when using (truncated) BPTT to train RNNs:\n",
        "\n",
        "* Shared / tied recurrent weights ($W_{hh}$) mean that **the gradient on these weights at some time step $t$ depends on all time steps up to time-step $T$**, the maximum length of the unrolled graph. This also leads to the **vanishing/exploding gradients** problem.\n",
        "\n",
        "* As alluded to above, **memory usage grows linearly with the total number of steps $T$ that we unroll for**, because we need to save/cache the activations at each time-step.  This matters computationally, since memory is a limited resource. It also matters statistically, because it puts a limit on the types of dependencies the model is exposed to, and hence that it could learn.\n",
        "\n",
        "**NOTE**: Think about that last statement and make sure you understand those 2 points.\n",
        "\n",
        "BPTT is very similar to the standard back-propagation algorithm. Key to understanding the BPTT algorithm is to realize that gradients on the non-recurrent weights (weights of a per time-step classifier that tries to predict the next word in a sentence for example) and recurrent weights (that transform $h_{t-1}$ into $h_t$) are computed differently:\n",
        "\n",
        "* The gradients of **non-recurrent weights** ($W_{hy}$) depend only on the error at that time-step, $E_t$.\n",
        "* The gradients of **recurrent weights** ($W_{hh}$) depend on all time-steps up to maximum length $T$.\n",
        "\n",
        "The first point is fairly intuitive: predictions at time-step $t$ is related to the loss of that particular prediction. \n",
        "\n",
        "The second point will be explained in more detail in the lectures (see also [this great blog post](http://www.wildml.com/2015/10/recurrent-neural-networks-tutorial-part-3-backpropagation-through-time-and-vanishing-gradients/)), but briefly, this can be summarized in these equations:\n",
        "\n",
        "1. The **current** state is a function of the **previous** state: $h_t = \\sigma(W_{hh}h_{t-1} + W_{xh}x_t)$\n",
        "2. The gradient of the loss $E_t$ at time $t$ on $W_{hh}$ is a function of the current hidden state and model predictions $\\hat{y}_t$ at time t: \n",
        "$\\frac{\\partial E_t}{\\partial W_{hh}} = \\frac{\\partial E_t}{\\partial \\hat{y}_t}\\frac{\\partial\\hat{y}_t}{\\partial h_t}\\frac{\\partial h_t}{\\partial W_{hh}}$\n",
        "3. Substituting (1) into (2) results in a **sum over all previous time-steps**:\n",
        "$\\frac{\\partial E_t}{\\partial W_{hh}} = \\sum\\limits_{k=0}^{t} \\frac{\\partial E_t}{\\partial \\hat{y}_t}\\frac{\\partial\\hat{y}_t}{\\partial h_t}\\frac{\\partial h_t}{\\partial h_k}\\frac{\\partial h_k}{\\partial W_{hh}}$\n",
        "\n",
        "Because of this **repeated multiplicative interaction**, as the sequence length $t$ gets longer, the gradients themselves can get diminishingly small (**vanish**) or grow too large and result in numeric overflow (**explode**). This has been shown to be related to the norms of the recurrent weight matrices being less than or equal to 1. Intuitively, it works very similar to how multiplying a small number $v<1.0$ with itself repeatedly can quickly go to zero, or conversely, a large number $v>1.0$ could quickly go to infinity; only this is for matrices.\n"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {
        "id": "9KRWSWRbGzBJ",
        "colab_type": "text"
      },
      "source": [
        "To implement this, we need three components:\n",
        "\n",
        "* The code to fprop one time-step though the cell,\n",
        "* the code to fprop through the unrolled RNN,\n",
        "* the code to backprop one time-step through the cell, \n",
        "\n",
        "And then we'll put all this together within the BPTT algorithm. Let's start.\n",
        "\n",
        "### Forward Propagation\n",
        "\n",
        "Let's write the code to fprop one time-step though the cell. We'll need some helper functions:"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {
        "id": "NINCPVs72WqW",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "import numpy as np\n",
        "\n",
        "## HELPER DEFINITIONS \n",
        "## NOTE: WE KEEP THESE EXPLICIT BECAUSE WE WILL NEED THEIR DERIVATIVES BELOW.\n",
        "\n",
        "def softmax(X):\n",
        "  eX = np.exp((X.T - np.max(X, axis=1)).T)\n",
        "  return (eX.T / eX.sum(axis=1)).T\n",
        "\n",
        "def cross_entropy(y_pred, y_train):\n",
        "  m = y_pred.shape[0]\n",
        "\n",
        "  prob = softmax(y_pred)\n",
        "  log_like = -np.log(prob[range(m), y_train])\n",
        "\n",
        "  data_loss = np.sum(log_like) / m\n",
        "\n",
        "  return data_loss\n",
        "\n",
        "def fc_forward(X, W, b):\n",
        "  '''A fully-connected feedforward layer.'''\n",
        "  out = np.dot(X, W) + b\n",
        "  cache = (W, X)\n",
        "  return out, cache\n",
        "\n",
        "def tanh_forward(X):\n",
        "  out = np.tanh(X)\n",
        "  cache = out\n",
        "  return out, cache"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "kikiBXVjIwG8",
        "colab_type": "text"
      },
      "source": [
        "Now we can implement the equation $h_t = \\sigma(W_{hh}h_{t-1} + W_{xh}x_t)$ (where $\\sigma$ is our non-linearity, tanh in this case) as follows:"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {
        "id": "Bw94oOd1I1Jj",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "## Define the RNN Cell. We use a vanilla RNN.\n",
        "def cell_fn_forward(X, h, model, train=True):\n",
        "  Wxh, Whh, Why = model['Wxh'], model['Whh'], model['Why']\n",
        "  bh, by = model['bh'], model['by']\n",
        "\n",
        "  hprev = h.copy()\n",
        "\n",
        "  h, h_cache = tanh_forward(np.dot(hprev, Whh) + np.dot(X, Wxh) + bh)\n",
        "  y, y_cache = fc_forward(h, Why, by)\n",
        "\n",
        "  cache = (X, Whh, h, hprev, y, h_cache, y_cache)\n",
        "\n",
        "  if not train:\n",
        "      # Compute per-time step outputs.\n",
        "      # NOTE: Here we build a classifer, but it could be anything else.\n",
        "      y = softmax(y)\n",
        "\n",
        "  return y, h, cache\n"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "TRFfMGOVTt-C",
        "colab_type": "text"
      },
      "source": [
        "Put this together to do the RNN fprop over the entire sequence:\n",
        "\n",
        "**QUESTION**: Notice how we save all activations in `caches`. Why do we need to do this?"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {
        "id": "XQUTNuJYTxqM",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "def rnn_forward(X_train, y_train, model, initial_state, verbose=True):\n",
        "  ys = []\n",
        "  caches = []\n",
        "  loss = 0.\n",
        "\n",
        "  h = initial_state\n",
        "  t = 0\n",
        "  for x, y in zip(X_train, y_train):\n",
        "    y_pred, h, cache = cell_fn_forward(x, h, model, train=True)\n",
        "    loss += cross_entropy(y_pred, y)\n",
        "    ys.append(y_pred)\n",
        "    caches.append(cache)\n",
        "    \n",
        "    if verbose:\n",
        "      print \"Time-step: \", t\n",
        "      print \"x_t = \", x\n",
        "      print \"cur_state = \", h\n",
        "      print \"predicted y = \", y_pred\n",
        "      \n",
        "    t += 1\n",
        "  \n",
        "  # We return final hidden state, predictions, caches and final total loss.\n",
        "  return h, ys, caches, loss\n"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "WLVIHIz1Kvug",
        "colab_type": "text"
      },
      "source": [
        "Let's test this on some dummy data."
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {
        "id": "NQJ7DH-gJxbq",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "output_extras": [
            {}
          ],
          "height": 731
        },
        "outputId": "ef70162d-0e2d-4316-bf85-e8efcdd55b5b",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1503667515621,
          "user_tz": -60,
          "elapsed": 536,
          "user": {
            "displayName": "Stephan Gouws",
            "photoUrl": "//lh4.googleusercontent.com/-6znVyM1oxdg/AAAAAAAAAAI/AAAAAAAAABI/vEPo2Ce7Rpc/s50-c-k-no/photo.jpg",
            "userId": "102606466886131565871"
          }
        }
      },
      "source": [
        "# Create a helper function that calculates the relative error between two arrays\n",
        "def relative_error(x, y):\n",
        "  \"\"\" returns relative error \"\"\"\n",
        "  return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))\n",
        "\n",
        "# We set the seed to make the results reproducible.\n",
        "np.random.seed(1234)\n",
        "\n",
        "def _initial_state(hidden_dim):\n",
        "  return np.zeros((1, hidden_dim))\n",
        "\n",
        "def _init_model(input_dim, hidden_dim, output_dim):\n",
        "  D, H, C = input_dim, hidden_dim, output_dim # More compact.\n",
        "  model_params = dict(\n",
        "    Wxh=np.random.randn(D, H) / np.sqrt(D / 2.),\n",
        "    Whh=np.random.randn(H, H) / np.sqrt(H / 2.),\n",
        "    Why=np.random.randn(H, D) / np.sqrt(C / 2.),\n",
        "    bh=np.zeros((1, H)),\n",
        "    by=np.zeros((1, D)))\n",
        "  return model_params\n",
        "\n",
        "\n",
        "# Initialize model.\n",
        "input_dim=8\n",
        "hidden_dim=4\n",
        "output_dim=2  # num_classes\n",
        "num_steps = 5\n",
        "\n",
        "test_mdl = _init_model(input_dim, hidden_dim, output_dim)\n",
        "\n",
        "# Create some dummy data (there's no batching, just num_steps input vectors)\n",
        "X_test = np.split(np.random.randn(num_steps, input_dim), num_steps, axis=0)\n",
        "y_test = np.random.randint(low=0, high=output_dim, size=num_steps).reshape(-1)\n",
        "#y_onehot = np.eye(output_dim)[y_ids]\n",
        "\n",
        "print \"Created dummy input data X\"\n",
        "print \"Created fake targets: \\n\", y_test\n",
        "#print \"Created fake onehot targets: \\n\", y_onehot\n",
        "\n",
        "print \"\\nRunning FPROP on dummy data: \"\n",
        "\n",
        "initial_state = _initial_state(hidden_dim)\n",
        "last_state, ys, caches, loss = rnn_forward(X_test, y_test, test_mdl, initial_state, verbose=True)\n",
        "  \n",
        "print \"Final hidden state: \", last_state\n",
        "\n",
        "correct_final_state = np.array([[ 0.19868001,  0.98286478,  0.76491549, -0.91578737]])\n",
        "\n",
        "# Compare your output to the \"correct\" ones \n",
        "# The difference should be around 2e-8 (or lower)\n",
        "print \"\\n============================\"\n",
        "print 'Testing rnn_forward'\n",
        "diff = relative_error(last_state, correct_final_state)\n",
        "if diff <= 2e-8:\n",
        "  print 'PASSED'\n",
        "else:\n",
        "  print 'The difference of %s is too high, try again' % diff\n",
        "print \"\\n============================\""
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Created dummy input data X\n",
            "Created fake targets: \n",
            "[1 0 0 1 0]\n",
            "\n",
            "Running FPROP on dummy data: \n",
            "Time-step:  0\n",
            "x_t =  [[ 0.46439233 -3.56351666  1.32110562  0.15263055  0.16452954 -0.43009569\n",
            "   0.76736874  0.98491984]]\n",
            "cur_state =  [[ 0.96177185 -0.9949211  -0.32244779  0.73506912]]\n",
            "predicted y =  [[ 0.55862327  1.40120743 -1.08963098 -1.08400572 -0.28057832 -1.1015924\n",
            "  -0.11587845 -0.05159674]]\n",
            "Time-step:  1\n",
            "x_t =  [[ 0.27083585  1.39198619  0.07984231 -0.39996458 -1.02785056 -0.58471821\n",
            "   0.81659393 -0.08194705]]\n",
            "cur_state =  [[ 0.46602766  0.82754901  0.36055494 -0.95185866]]\n",
            "predicted y =  [[ 0.34742921 -1.29286866  2.47391689 -1.71784312 -1.5992273   1.24601455\n",
            "  -0.88538448 -0.11095221]]\n",
            "Time-step:  2\n",
            "x_t =  [[-0.34476601  0.52828815 -1.06898878 -0.51188131  0.29120536  0.5665337\n",
            "   0.50359176  0.28529568]]\n",
            "cur_state =  [[ 0.05420772  0.99026303  0.59787223 -0.29612811]]\n",
            "predicted y =  [[ 0.8010615  -1.61368801  2.09415036  0.17450844 -0.9812788   0.68753718\n",
            "  -0.38148559  0.05024376]]\n",
            "Time-step:  3\n",
            "x_t =  [[ 0.48428811  1.36348151 -0.78110528 -0.46801767  1.22457436 -1.28110828\n",
            "   0.8754755  -1.71071532]]\n",
            "cur_state =  [[ 0.54636226  0.72291255  0.98786587 -0.98647387]]\n",
            "predicted y =  [[ 0.99520914 -2.79581132  3.78816447 -2.78201096 -1.48485321  1.72724645\n",
            "  -1.47624028  0.1877338 ]]\n",
            "Time-step:  4\n",
            "x_t =  [[-0.4507651   0.74916381 -0.20393287 -0.18217541  0.680656   -1.81849899\n",
            "   0.04707164  0.39484421]]\n",
            "cur_state =  [[ 0.19868001  0.98286478  0.76491549 -0.91578737]]\n",
            "predicted y =  [[ 0.64388666 -2.30770109  3.12462806 -1.48085355 -1.28186319  1.50545244\n",
            "  -0.98578343  0.08359233]]\n",
            "Final hidden state:  [[ 0.19868001  0.98286478  0.76491549 -0.91578737]]\n",
            "\n",
            "============================\n",
            "Testing rnn_forward\n",
            "PASSED\n",
            "\n",
            "============================\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "H3q0dGyzG6Ji",
        "colab_type": "text"
      },
      "source": [
        "### Computing the derivative: Truncated BPTT \n",
        "\n",
        "Let's start with computing the per time-step derivative of `recurrent_fn` wrt all model parameters. First, some helper derivative functions:\n",
        "\n",
        "**NOTE**: Make sure you understand how these were derived."
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {
        "id": "rTCynUOYEg5i",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "## HELPER DERIVATIVE FUNCTIONS\n",
        "def fc_backward(dout, cache):\n",
        "  W, h = cache\n",
        "\n",
        "  dW = np.dot(h.T, dout)\n",
        "  db = np.sum(dout, axis=0)\n",
        "  dX = np.dot(dout, W.T)\n",
        "\n",
        "  return dX, dW, db\n",
        "\n",
        "def tanh_backward(dout, cache):\n",
        "  dX = (1 - cache**2) * dout\n",
        "  return dX\n",
        "\n",
        "def dcross_entropy(y_pred, y_train):\n",
        "  m = y_pred.shape[0]\n",
        "  \n",
        "  grad_y = softmax(y_pred)\n",
        "  grad_y[range(m), y_train] -= 1.\n",
        "  grad_y /= m\n",
        "\n",
        "  return grad_y"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "evUka0sdM6ul",
        "colab_type": "text"
      },
      "source": [
        "Let's put these together and write the code for $\\frac{\\partial E}{\\partial \\theta}$ for all parameters:"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {
        "id": "wo2xTk99M76j",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "## PERFORM PER-TIMESTEP BACKWARD STEP\n",
        "\n",
        "def cell_fn_backward(y_pred, y_train, dh_next, cache):\n",
        "  X, Whh, h, hprev, y, h_cache, y_cache = cache\n",
        "\n",
        "  # Softmax gradient\n",
        "  dy = dcross_entropy(y_pred, y_train)\n",
        "\n",
        "  # Hidden to output gradient\n",
        "  dh, dWhy, dby = fc_backward(dy, y_cache)\n",
        "  dh += dh_next\n",
        "  dby = dby.reshape((1, -1))\n",
        "\n",
        "  # tanh\n",
        "  dh = tanh_backward(dh, h_cache)\n",
        "\n",
        "  # Hidden gradient\n",
        "  dbh = dh\n",
        "  dWhh = np.dot(hprev.T, dh)\n",
        "  dWxh = np.dot(X.T, dh)\n",
        "  dh_next = np.dot(dh, Whh.T)\n",
        "\n",
        "  grad = dict(Wxh=dWxh, Whh=dWhh, Why=dWhy, bh=dbh, by=dby)\n",
        "\n",
        "  return grad, dh_next"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "-6hdtyrPOCvH",
        "colab_type": "text"
      },
      "source": [
        "Now let's put this together inside the BPTT algorithm:"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {
        "id": "CHVwqUHvEmfa",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "def bptt(model, X_train, y_train, initial_state):\n",
        "  # Forward\n",
        "  last_state, ys, caches, loss = rnn_forward(X_train, y_train, model, initial_state)\n",
        "\n",
        "  loss /= y_train.shape[0]\n",
        "\n",
        "  # Backward\n",
        "  dh_next = np.zeros((1, last_state.shape[0]))\n",
        "  grads = {k: np.zeros_like(v) for k, v in model.items()}\n",
        "\n",
        "  for t in reversed(range(len(X_train))):\n",
        "    grad, dh_next = cell_fn_backward(ys[t], y_train[t], dh_next, caches[t])\n",
        "\n",
        "    for k in grads.keys():\n",
        "        grads[k] += grad[k]\n",
        "\n",
        "  for k, v in grads.items():\n",
        "    grads[k] = np.clip(v, -5., 5.)\n",
        "\n",
        "  return grads, loss, last_state\n"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "55st2mH8ghcV",
        "colab_type": "text"
      },
      "source": [
        "Finally, we can check our implementation using numerically-derived gradients:"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {
        "id": "eNSXgrwvgtYY",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "def rnn_gradient_check(model, x, y, init_state, h=0.001, error_threshold=0.01):\n",
        "  \n",
        "  # Calculate the gradients using backpropagation. We want to checker if these are correct.\n",
        "  bptt_gradients, _, _ = bptt(model, x, y, init_state)\n",
        "  \n",
        "  # List of all parameters we want to check.\n",
        "  model_parameters = ['Wxh', 'Whh', 'Why', 'bh', 'by']\n",
        "  \n",
        "  # Gradient check for each parameter\n",
        "  for pidx, pname in enumerate(model_parameters):\n",
        "    # Get the actual parameter value from the model, e.g. model.W\n",
        "    parameter = model[pname]\n",
        "    \n",
        "    print \"Performing gradient check for parameter %s with size %d.\" % (pname, np.prod(parameter.shape))\n",
        "    \n",
        "    # Iterate over each element of the parameter matrix, e.g. (0,0), (0,1), ...\n",
        "    it = np.nditer(parameter, flags=['multi_index'], op_flags=['readwrite'])\n",
        "    \n",
        "    while not it.finished:\n",
        "      \n",
        "      ix = it.multi_index\n",
        "      # Save the original value so we can reset it later\n",
        "      original_value = parameter[ix]\n",
        "      \n",
        "      # Estimate the gradient using (f(x+h) - f(x-h))/(2*h)\n",
        "      parameter[ix] = original_value + h\n",
        "      # Compute the ENTIRE rnn_foward, evaluate cross entropy loss\n",
        "      _, _, _, gradplus = rnn_forward(x, y, model, init_state, verbose=False)\n",
        "      parameter[ix] = original_value - h\n",
        "      _, _, _, gradminus = rnn_forward(x, y, model, init_state, verbose=False)\n",
        "      estimated_gradient = (gradplus - gradminus)/(2*h)\n",
        "      \n",
        "      # Reset parameter to original value\n",
        "      parameter[ix] = original_value\n",
        "      \n",
        "      # The gradient for this parameter calculated using backpropagation\n",
        "      backprop_gradient = bptt_gradients[pname][ix]\n",
        "      \n",
        "      # calculate The relative error: (|x - y|/(|x| + |y|))\n",
        "      relative_error = np.abs(backprop_gradient - estimated_gradient) / (np.abs(backprop_gradient) + np.abs(estimated_gradient))\n",
        "      \n",
        "      # If the error is to large fail the gradient check\n",
        "      if relative_error > error_threshold:\n",
        "        print \"Gradient Check ERROR: parameter=%s ix=%s\" % (pname, ix)\n",
        "        print \"+h Loss: %f\" % gradplus\n",
        "        print \"-h Loss: %f\" % gradminus\n",
        "        print \"Estimated_gradient: %f\" % estimated_gradient\n",
        "        print \"Backpropagation gradient: %f\" % backprop_gradient\n",
        "        print \"Relative Error: %f\" % relative_error\n",
        "        return\n",
        "      it.iternext()\n",
        "    \n",
        "    print \"Gradient check for parameter %s: PASSED.\" % (pname)\n"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "CtGlXaAZkEpV",
        "colab_type": "text"
      },
      "source": [
        "Aaaaand let's test it!"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {
        "id": "yPt9o7pwkEVE",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "output_extras": [
            {}
          ],
          "height": 697
        },
        "outputId": "7137ac0e-93d0-4ea7-c9b7-96ccb1751379",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1503668261812,
          "user_tz": -60,
          "elapsed": 410,
          "user": {
            "displayName": "Stephan Gouws",
            "photoUrl": "//lh4.googleusercontent.com/-6znVyM1oxdg/AAAAAAAAAAI/AAAAAAAAABI/vEPo2Ce7Rpc/s50-c-k-no/photo.jpg",
            "userId": "102606466886131565871"
          }
        }
      },
      "source": [
        "rnn_gradient_check(test_mdl, X_test, y_test, initial_state)"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Time-step:  0\n",
            "x_t =  [[ 0.46439233 -3.56351666  1.32110562  0.15263055  0.16452954 -0.43009569\n",
            "   0.76736874  0.98491984]]\n",
            "cur_state =  [[ 0.96177185 -0.9949211  -0.32244779  0.73506912]]\n",
            "predicted y =  [[ 0.55862327  1.40120743 -1.08963098 -1.08400572 -0.28057832 -1.1015924\n",
            "  -0.11587845 -0.05159674]]\n",
            "Time-step:  1\n",
            "x_t =  [[ 0.27083585  1.39198619  0.07984231 -0.39996458 -1.02785056 -0.58471821\n",
            "   0.81659393 -0.08194705]]\n",
            "cur_state =  [[ 0.46602766  0.82754901  0.36055494 -0.95185866]]\n",
            "predicted y =  [[ 0.34742921 -1.29286866  2.47391689 -1.71784312 -1.5992273   1.24601455\n",
            "  -0.88538448 -0.11095221]]\n",
            "Time-step:  2\n",
            "x_t =  [[-0.34476601  0.52828815 -1.06898878 -0.51188131  0.29120536  0.5665337\n",
            "   0.50359176  0.28529568]]\n",
            "cur_state =  [[ 0.05420772  0.99026303  0.59787223 -0.29612811]]\n",
            "predicted y =  [[ 0.8010615  -1.61368801  2.09415036  0.17450844 -0.9812788   0.68753718\n",
            "  -0.38148559  0.05024376]]\n",
            "Time-step:  3\n",
            "x_t =  [[ 0.48428811  1.36348151 -0.78110528 -0.46801767  1.22457436 -1.28110828\n",
            "   0.8754755  -1.71071532]]\n",
            "cur_state =  [[ 0.54636226  0.72291255  0.98786587 -0.98647387]]\n",
            "predicted y =  [[ 0.99520914 -2.79581132  3.78816447 -2.78201096 -1.48485321  1.72724645\n",
            "  -1.47624028  0.1877338 ]]\n",
            "Time-step:  4\n",
            "x_t =  [[-0.4507651   0.74916381 -0.20393287 -0.18217541  0.680656   -1.81849899\n",
            "   0.04707164  0.39484421]]\n",
            "cur_state =  [[ 0.19868001  0.98286478  0.76491549 -0.91578737]]\n",
            "predicted y =  [[ 0.64388666 -2.30770109  3.12462806 -1.48085355 -1.28186319  1.50545244\n",
            "  -0.98578343  0.08359233]]\n",
            "Performing gradient check for parameter Wxh with size 32.\n",
            "Gradient check for parameter Wxh: PASSED.\n",
            "Performing gradient check for parameter Whh with size 16.\n",
            "Gradient check for parameter Whh: PASSED.\n",
            "Performing gradient check for parameter Why with size 32.\n",
            "Gradient check for parameter Why: PASSED.\n",
            "Performing gradient check for parameter bh with size 4.\n",
            "Gradient check for parameter bh: PASSED.\n",
            "Performing gradient check for parameter by with size 8.\n",
            "Gradient check for parameter by: PASSED.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "tYTNSQwQfpE7",
        "colab_type": "text"
      },
      "source": [
        "# Moving on to MNIST data"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {
        "id": "IyIHQpUGgXyM",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "output_extras": [
            {}
          ],
          "height": 85
        },
        "outputId": "f77b737f-d410-4a1a-fb44-2fdb96ff8f7c",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1503667860083,
          "user_tz": -60,
          "elapsed": 7545,
          "user": {
            "displayName": "Stephan Gouws",
            "photoUrl": "//lh4.googleusercontent.com/-6znVyM1oxdg/AAAAAAAAAAI/AAAAAAAAABI/vEPo2Ce7Rpc/s50-c-k-no/photo.jpg",
            "userId": "102606466886131565871"
          }
        }
      },
      "source": [
        "import tensorflow as tf\n",
        "\n",
        "import numpy as np\n",
        "import functools\n",
        "\n",
        "from tensorflow.examples.tutorials.mnist import input_data\n",
        "\n",
        "\n",
        "mnist = input_data.read_data_sets(\"MNIST_data/\", one_hot=True)\n"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Extracting MNIST_data/train-images-idx3-ubyte.gz\n",
            "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n",
            "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n",
            "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "esdB5QtJFixP",
        "colab_type": "text"
      },
      "source": [
        "# Gated Cells (GRUs and LSTMs)\n",
        "\n",
        "Vanilla RNNs are very powerful sequence models. However, they are difficult to train, in large part due to the difficulties with getting gradients to propagate through all the time-steps without vanishing or exploding. If the gradient explodes then backpropagation will not work because we will get NaN values for the gradient contributions from earlier layers. The simplest trick to overcome this is called **gradient clipping** (see [Pascanu et al., 2013](http://www.jmlr.org/proceedings/papers/v28/pascanu13.pdf)). One basically rescales the gradients once their norms exceed a certain threshold. Dealing with vanishing gradients is trickier. Proper weight initialization helps to overcome this at the start of training (e.g [orthogonal initialization](https://www.tensorflow.org/api_docs/python/tf/orthogonal_initializer)), and there are regularization tricks for encouraging constant backwards error flow, which works for some tasks, but is not theoretically well-motivated.\n",
        "\n",
        "Gated models (we will look at the GRU and LSTM) modify the architecture of the RNN cell to ensure constant gradient propagation. The problem with the vanilla RNN is that its entire state is multiplicatively updated (overwritten) at every time-step (notice the $W_{hh}h_{t-1}$ term): \n",
        "\n",
        "$h_t = \\sigma(W_{hh}h_{t-1} ...)$\n",
        "\n",
        "Gated cells (GRUs/LSTMs) have two main new ideas:\n",
        "\n",
        "* Ensure **incremental state change** by updating state *additively*: $h_t = h_{t-1} + f(h_{t-1})$.\n",
        "* Control the update process by selectively **modulating** how much to keep/forget of the old, how much to read, and how much to write into the new state.\n",
        "\n",
        "These models modulate how much to throw away (forget) from the previous state when making a proposed new state, and then how much to read from the previous state and write to the output of the cell, by using **gates** (values between 0 and 1 which squash information flow to some extent; little neural networks, of course!). Gates are  vectors of *per-dimension interpolation scalars*: When you multiply some vector by a gate vector, you essentially control how much of that vector you \"let through\".  Below we show the generic equation for such a gate (they're all the same!):\n",
        "\n",
        "$g_t = \\sigma(W_g h_{t-1} + U_g x_t + b_g)$\n",
        "\n",
        "where $\\sigma(z) = 1 / (1+e^{-z})$ is the sigmoid function (i.e. $0 \\leq \\sigma(z) \\leq 1.$). \n",
        "\n",
        "Think about this for a second:\n",
        "\n",
        "* What are the inputs of this gate (model)?\n",
        "* What are the parameters of this gate (model)?\n",
        "* What does this remind you of? \n",
        "\n",
        "**NOTE**: Gates are just (vectors of) simple, logistic regression models which take inputs from the previous hidden layer $h_{t-1}$ and the current input layer $x_t$) and produce outputs between 0 and 1. \n",
        "\n",
        "Now let's use them to modulate the flow of information. We'll start with the GRU and build up to the LSTM.\n",
        "\n",
        "\n",
        "## The Gated Recurrent Unit (GRU)\n",
        "\n",
        "The GRU was introduced in 2014 by Cho et al. -- almost 2 decades after the LSTM -- but we'll start here because the GRU is simpler than the LSTM. It uses only two gates per cell. First, it uses a **reset** gate to control how much of the previous state is used in computing the new proposed state:\n",
        "\n",
        "$\\tilde{h_t} = \\phi(W(r_t \\bigodot h_{t-1}) + Ux_t + b)$\n",
        "\n",
        "(where $\\bigodot$ is element-wise multiplication). Then it ties \"reading\" and \"writing\" into an **update** gate (by bounding their sums to 1, i.e. the more you read the less you write) when the new state is calculated:\n",
        "\n",
        "$h_t = z_t \\bigodot h_{t-1} + (1 - z_t)\\bigodot \\tilde{h_t}$\n",
        "\n",
        "Try to reconcile the equations with the following flow diagram of the same:\n",
        "\n",
        "**TODO**: Include the picture for this: https://r2rt.com/static/images/NH_GRUCell.png \n",
        "\n",
        "* What happens when the reset gate is high/low?\n",
        "* What happens when the update gate is high/low?\n",
        "* How do these two interact?\n",
        "* Why would this architecture be more powerful than a vanilla RNN?\n",
        "\n",
        "## The Long Short-Term Memory unit (LSTM)\n",
        "\n",
        "The LSTM was introduced in 1997 by Hochreiter and Schmidhuber. There are several different architectual variations 'out there', but they all operate by maintaining a separate **memory vector** $c_t$ and a **state vector** $h_t$ (i.e. the model computes a tuple of vectors per time-step, not just a single vector). We'll just focus on the `BasicLSTM` version now. It uses three gates, the **input**, **output** and **forget** gates:\n",
        "\n",
        "\\begin{aligned}\n",
        " i_t &= \\sigma(W_i h_{t-1} + U_i x_t + b_i) \\\\\n",
        " o_t &= \\sigma(W_o h_{t-1} + U_o x_t + b_o) \\\\\n",
        " f_t &= \\sigma(W_f h_{t-1} + U_f x_t + b_f) \\\\\n",
        "\\end{aligned}\n",
        "\n",
        "Don't be intimidated by these equations. We've seen them all above already in their generic form. They're all just doing the same thing (computationally, not functionally). Convince yourself of this, by answering the following:\n",
        "\n",
        "* What is the **same** between these equations?\n",
        "* What is **different** between them?\n",
        "\n",
        "1. First, the `BasicLSTM` Cell uses no gating to create the proposed new hidden state (original notation uses g, but I use tilde h):\n",
        "\n",
        "      $\\tilde{h}_t = \\phi(W h_{t-1}) + Ux_t + b)$\n",
        "\n",
        "2. Then it updates its internal memory to be a combination of the previous memory $c_{t-1}$ (multiplied/modulated by the forget gates $f_t$) and the new proposed state $\\tilde{h}_t$ (modulated by the input gates $i_t$):\n",
        "\n",
        "      $c_t = f_t \\bigodot c_{t-1} + i_t \\bigodot \\tilde{h}_t$\n",
        "\n",
        "      **Intuitively**: the model could choose to ignore old memory completely (if $f_t$ is all 1s), or ignore the newly proposed state completely ($i_t$ all 0s), but more likely it would learn to do something in-between. Think about how and why this is true.\n",
        "\n",
        "3. Finally, the *state* that is actually output by the cell is a gated version of the memory vector, squashed by a tanh (because not everything in the memory cell might be immediately useful to the surrounding network):\n",
        "\n",
        "      $ h_t = o_t \\bigodot \\phi(c_t) $\n",
        "\n",
        "*Phewwww*. We know.. there is a lot going on here! The following flow-diagram might help a bit to make this more clear:\n",
        "\n",
        "**TODO** Add picture: https://r2rt.com/static/images/NH_BasicLSTMCell.png\n",
        "\n",
        "Look at the equations and at the flow-diagram, and then try to answer the following questions:\n",
        "\n",
        "* How is the LSTM similar to the GRU?\n",
        "* How is it different?\n",
        "* What is the function of the memory vector (think about edge cases, e.g. where the forget gate is set to all 1s)?\n",
        "* Is the LSTM theoretically more powerful than the GRU? If so, why?\n",
        "* What is the computational drawback to using LSTMs (think about the number of gates; these must be parameterized..)?"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {
        "id": "B_eGri3h8St8",
        "colab_type": "text"
      },
      "source": [
        "# Implementing Recurrent Models in TensorFlow\n",
        "\n",
        "In TensorFlow, we implement recurrent models using two building blocks:\n",
        "\n",
        " 1. A (graph) definition for the cell (you can use [those provided](https://www.tensorflow.org/versions/r1.0/api_guides/python/contrib.rnn#Base_interface_for_all_RNN_Cells), or you can write your own); and\n",
        " 2. A method to unroll the graph over the sequence (dynamic vs unrolled).\n",
        " \n",
        "First, TensorFlow provides implementations for many of the standard RNN cells. Poke around in the docs.\n",
        "\n",
        "Second, TensorFlow provides two different ways to implement the recurrence operations: **dynamic or static**. Basically, static unrolling prebuilds the entire unrolled RNN over the maximum number of time-steps. Dynamic unrolling dynamically creates the graph at each time-step, and saves the activations during the forward phase for the backward phase.\n",
        "\n",
        "**QUESTION**: Why do activations need to be saved during the forward phase? (HINT: Look at our use of `cache` in the Numpy code above).\n",
        "\n",
        "We will use dynamic unrolling: it uses less memory, and (counterintuitively), oftentimes it turns out to be faster."
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {
        "id": "-wnPo8X1gp1e",
        "colab_type": "text"
      },
      "source": [
        "# An RNN Image Labeler\n",
        "\n",
        "Let's reuse the BaseSoftmaxClassifier from the previous practicals:"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {
        "id": "9rk6sOnPf286",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "class BaseSoftmaxClassifier(object):\n",
        "  def __init__(self, input_size, output_size, l2_lambda):    \n",
        "    # Define the input placeholders. The \"None\" dimension means that the \n",
        "    # placeholder can take any number of images as the batch size. \n",
        "    self.x = tf.placeholder(tf.float32, [None, input_size], name='x')\n",
        "    self.y = tf.placeholder(tf.float32, [None, output_size], name='y')  \n",
        "    self.input_size = input_size\n",
        "    self.output_size = output_size\n",
        "    self.l2_lambda = l2_lambda\n",
        "\n",
        "    self._all_weights = [] # Used to compute L2 regularization in compute_loss().\n",
        "    \n",
        "    # You should override these in your build_model() function.\n",
        "    self.logits = None\n",
        "    self.predictions = None\n",
        "    self.loss = None\n",
        "    \n",
        "    self.build_model()\n",
        "    \n",
        "  def get_logits(self):\n",
        "    return self.logits\n",
        "  \n",
        "  def build_model(self):\n",
        "    # OVERRIDE THIS FOR YOUR PARTICULAR MODEL.\n",
        "    raise NotImplementedError(\"Subclasses should implement this function!\")\n",
        "    \n",
        "  def compute_loss(self):\n",
        "    \"\"\"All models share the same softmax cross-entropy loss.\"\"\"\n",
        "    assert self.logits is not None  # Ensure that logits has been created! \n",
        "    data_loss = tf.reduce_mean(\n",
        "        tf.nn.softmax_cross_entropy_with_logits(logits=self.logits, labels=self.y))\n",
        "    reg_loss = 0.\n",
        "    for w in self._all_weights:\n",
        "      reg_loss += tf.nn.l2_loss(w)\n",
        "      \n",
        "    return data_loss + self.l2_lambda * reg_loss\n",
        "  \n",
        "  def accuracy(self):\n",
        "    # Calculate accuracy.\n",
        "    assert self.predictions is not None  # Ensure that pred has been created!\n",
        "    correct_prediction = tf.equal(tf.argmax(self.predictions, 1), tf.argmax(self.y, 1))\n",
        "    accuracy = tf.reduce_mean(tf.cast(correct_prediction, \"float\"))\n",
        "    return accuracy"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ZNEifoNh8H58",
        "colab_type": "text"
      },
      "source": [
        "We override build_model to build the graph for the RNN classifier."
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {
        "id": "TUZ5BZUiggm9",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "class RecurrentClassifier(BaseSoftmaxClassifier):\n",
        "    def __init__(self, model_params):\n",
        "      \n",
        "        self.config = model_params\n",
        "        \n",
        "        super(RecurrentClassifier, self).__init__(model_params['input_size'],\n",
        "                                                  model_params['output_size'],\n",
        "                                                  model_params['l2_lambda'])\n",
        "       \n",
        "\n",
        "    def build_model(self):\n",
        "        \n",
        "        assert self.config['num_steps'] * self.config['pixels_per_step'] == self.config['input_size']\n",
        "        # We break up the input images into num_steps groups of pixels_per_step\n",
        "        # pixels each.    \n",
        "        rnn_input = tf.reshape(self.x, [-1, \n",
        "                                        self.config['num_steps'], \n",
        "                                        self.config['pixels_per_step']])\n",
        "\n",
        "        \n",
        "        # Define the main RNN 'cell', that will be applied to each timestep.\n",
        "        cell = self.config['cell_fn'](self.config['memory_units'])\n",
        "        # NOTE: This is how we apply Dropout to RNNs.\n",
        "        cell = tf.contrib.rnn.DropoutWrapper(\n",
        "            cell, \n",
        "            output_keep_prob = self.config['dropout_keep_prob'])\n",
        "        cell = tf.contrib.rnn.MultiRNNCell(cells=[cell] * self.config['num_layers'],\n",
        "                                           state_is_tuple=True)\n",
        "                \n",
        "        outputs, state = tf.nn.dynamic_rnn(cell, \n",
        "                                          rnn_input, \n",
        "                                          dtype=tf.float32)\n",
        "        \n",
        "        # Transpose the cell to get the output from the last timestep for each batch.\n",
        "        output = tf.transpose(outputs, [1, 0, 2])\n",
        "        last_hiddens = tf.gather(output, int(output.get_shape()[0]) - 1)\n",
        "        \n",
        "        # Define weights and biases for output prediction.\n",
        "        out_weights = tf.Variable(tf.random_normal([self.config['memory_units'],\n",
        "                                                   self.config['output_size']]))\n",
        "        self._all_weights.append(out_weights)\n",
        "        \n",
        "        out_biases = tf.Variable(tf.random_normal([self.config['output_size']]))\n",
        "        \n",
        "        self.logits = tf.matmul(last_hiddens, out_weights) + out_biases\n",
        "        \n",
        "        self.predictions = tf.nn.softmax(self.logits)\n",
        "        self.loss = self.compute_loss()\n",
        "        \n",
        "    \n",
        "    def get_logits(self):\n",
        "        return self.logits"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "wkk33_lk7Z-I",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "class MNISTFraction(object):\n",
        "  \"\"\"A helper class to extract only a fixed fraction of MNIST data.\"\"\"\n",
        "  def __init__(self, mnist, fraction):\n",
        "    self.mnist = mnist\n",
        "    self.num_images = int(mnist.num_examples * fraction)\n",
        "    self.image_data, self.label_data = mnist.images[:self.num_images], mnist.labels[:self.num_images]\n",
        "    self.start = 0\n",
        "    \n",
        "  def next_batch(self, batch_size):\n",
        "    start = self.start\n",
        "    end = min(start + batch_size, self.num_images)\n",
        "    self.start = 0 if end == self.num_images else end\n",
        "    return self.image_data[start:end], self.label_data[start:end]"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Jid1Ejbj7PO8",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "def train_tf_model(tf_model,                   \n",
        "                   session,  # The active session.\n",
        "                   num_epochs,  # Max epochs/iterations to train for.\n",
        "                   batch_size=50,  # Number of examples per batch.\n",
        "                   keep_prob=1.0,  # (1. - dropout) probability, none by default.\n",
        "                   train_only_on_fraction=1.,  # Fraction of training data to use.\n",
        "                   optimizer_fn=None,  # TODO(sgouws): more correct to call this optimizer_obj\n",
        "                   report_every=1, # Report training results every nr of epochs.\n",
        "                   eval_every=1,  # Evaluate on validation data every nr of epochs.\n",
        "                   stop_early=True,  # Use early stopping or not.\n",
        "                   verbose=True): \n",
        "\n",
        "  # Get the (symbolic) model input, output, loss and accuracy.\n",
        "  x, y = tf_model.x, tf_model.y\n",
        "  loss = tf_model.loss\n",
        "  accuracy = tf_model.accuracy()\n",
        "\n",
        "  # Compute the gradient of the loss with respect to the model parameters \n",
        "  # and create an op that will perform one parameter update using the specific\n",
        "  # optimizer's update rule in the direction of the gradients.\n",
        "  if optimizer_fn is None:\n",
        "    optimizer_fn = tf.train.AdamOptimizer()\n",
        "  optimizer_step = optimizer_fn.minimize(loss)\n",
        "\n",
        "  # Get the op which, when executed, will initialize the variables.\n",
        "  init = tf.global_variables_initializer()\n",
        "  # Actually initialize the variables (run the op).\n",
        "  session.run(init)\n",
        "\n",
        "  # Save the training loss and accuracies on training and validation data.\n",
        "  train_costs = []\n",
        "  train_accs = []\n",
        "  val_costs = []\n",
        "  val_accs = []\n",
        "\n",
        "  if train_only_on_fraction < 1:\n",
        "    mnist_train_data = MNISTFraction(mnist.train, train_only_on_fraction)\n",
        "  else:\n",
        "    mnist_train_data = mnist.train\n",
        "  \n",
        "  prev_c_eval = 1000000\n",
        "  \n",
        "  # Main training cycle.\n",
        "  for epoch in range(num_epochs):\n",
        "\n",
        "    avg_cost = 0.\n",
        "    avg_acc = 0.\n",
        "    total_batch = int(train_only_on_fraction * mnist.train.num_examples / batch_size)\n",
        "\n",
        "    # Loop over all batches.\n",
        "    for i in range(total_batch):\n",
        "      batch_x, batch_y = mnist_train_data.next_batch(batch_size)\n",
        "              \n",
        "      # Run optimization op (backprop) and cost op (to get loss value),\n",
        "      # and compute the accuracy of the model.\n",
        "      feed_dict = {x: batch_x, y: batch_y}\n",
        "      if keep_prob < 1.:\n",
        "        feed_dict[\"keep_prob:0\"] = keep_prob\n",
        "        \n",
        "      _, c, a = session.run(\n",
        "          [optimizer_step, loss, accuracy], feed_dict=feed_dict)\n",
        "            \n",
        "      # Compute average loss/accuracy\n",
        "      avg_cost += c / total_batch\n",
        "      avg_acc += a / total_batch      \n",
        "    \n",
        "    train_costs.append((epoch, avg_cost))\n",
        "    train_accs.append((epoch, avg_acc))\n",
        "\n",
        "    # Display logs per epoch step\n",
        "    if epoch % report_every == 0 and verbose:\n",
        "      print \"Epoch:\", '%04d' % (epoch+1), \"Training cost =\", \\\n",
        "            \"{:.9f}\".format(avg_cost)\n",
        "        \n",
        "    if epoch % eval_every == 0:\n",
        "      val_x, val_y = mnist.validation.images, mnist.validation.labels      \n",
        "      \n",
        "      feed_dict = {x : val_x, y : val_y}\n",
        "      if keep_prob < 1.:\n",
        "        feed_dict['keep_prob:0'] = 1.0\n",
        "        \n",
        "      c_eval, a_eval = session.run([loss, accuracy], feed_dict=feed_dict)\n",
        "      \n",
        "      if verbose:\n",
        "        print \"Epoch:\", '%04d' % (epoch+1), \"Validation acc=\", \\\n",
        "              \"{:.9f}\".format(a_eval)\n",
        "        \n",
        "      if c_eval >= prev_c_eval and stop_early:\n",
        "        print \"Validation loss stopped improving, stopping training early after %d epochs!\" % (epoch + 1)\n",
        "        break\n",
        "        \n",
        "      prev_c_eval = c_eval\n",
        "        \n",
        "      val_costs.append((epoch, c_eval))\n",
        "      val_accs.append((epoch, a_eval))\n",
        "      \n",
        "  \n",
        "  print \"Optimization Finished!\"\n",
        "  return train_costs, train_accs, val_costs, val_accs\n"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "5Gnj1bC67dwT",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "# Helper functions to plot training progress.\n",
        "\n",
        "from matplotlib import pyplot as plt\n",
        "\n",
        "def my_plot(list_of_tuples):\n",
        "  \"\"\"Take a list of (epoch, value) and split these into lists of \n",
        "  epoch-only and value-only. Pass these to plot to make sure we\n",
        "  line up the values at the correct time-steps.\n",
        "  \"\"\"\n",
        "  plt.plot(*zip(*list_of_tuples))\n",
        "\n",
        "def plot_multi(values_lst, labels_lst, y_label, x_label='epoch'):\n",
        "  # Plot multiple curves.\n",
        "  assert len(values_lst) == len(labels_lst)\n",
        "  plt.subplot(2, 1, 2)\n",
        "  \n",
        "  for v in values_lst:\n",
        "    my_plot(v)\n",
        "  plt.legend(labels_lst, loc='upper left')\n",
        "  \n",
        "  plt.xlabel(x_label)\n",
        "  plt.ylabel(y_label)\n",
        "  plt.show()  \n"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "SPfywldo94TN",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "output_extras": [
            {},
            {},
            {},
            {}
          ],
          "height": 856
        },
        "outputId": "d78319c0-0d58-43fe-ef57-8fca5dabeed7",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1503339496820,
          "user_tz": -60,
          "elapsed": 235339,
          "user": {
            "displayName": "Stephan Gouws",
            "photoUrl": "//lh4.googleusercontent.com/-6znVyM1oxdg/AAAAAAAAAAI/AAAAAAAAABI/vEPo2Ce7Rpc/s50-c-k-no/photo.jpg",
            "userId": "102606466886131565871"
          }
        }
      },
      "source": [
        "%%time\n",
        "\n",
        "def build_train_eval_and_plot(model_params, train_params, verbose=True):\n",
        "  \n",
        "  tf.reset_default_graph()\n",
        "  m = Recurrentlassifier(model_params)\n",
        "\n",
        "  with tf.Session() as sess:\n",
        "    # Train model on the MNIST dataset.\n",
        "   \n",
        "    train_losses, train_accs, val_losses, val_accs = train_tf_model(\n",
        "        m, \n",
        "        sess,\n",
        "        verbose=verbose,\n",
        "        **train_params) \n",
        "    \n",
        "    # Now evaluate it on the test set:\n",
        "  \n",
        "    accuracy_op = m.accuracy()  # Get the symbolic accuracy operation\n",
        "    # Calculate the accuracy using the test images and labels.\n",
        "    accuracy = accuracy_op.eval({m.x: mnist.test.images, \n",
        "                                 m.y: mnist.test.labels})  \n",
        "    \n",
        "    if verbose: \n",
        "      print \"Accuracy on test set:\", accuracy\n",
        "      # Plot losses and accuracies.\n",
        "      plot_multi([train_losses, val_losses], ['train', 'val'], 'loss', 'epoch')\n",
        "      plot_multi([train_accs, val_accs], ['train', 'val'], 'accuracy', 'epoch')\n",
        "      \n",
        "    \n",
        "    ret = {'train_losses': train_losses, 'train_accs' : train_accs,\n",
        "           'val_losses' : val_losses, 'val_accs' : val_accs,\n",
        "           'test_acc' : accuracy}\n",
        "    \n",
        "    return m, ret\n",
        "\n",
        "#################################CODE TEMPLATE##################################\n",
        "# Specify the model hyperparameters:\n",
        "model_params = {\n",
        "    'input_size' : 784,\n",
        "    'output_size' : 10,\n",
        "    'batch_size'  : 100,\n",
        "    'num_steps' : 28,          \n",
        "    'pixels_per_step' : 28,  # NOTE: num_steps * pixels_per_step must = input_size\n",
        "    'cell_fn' : tf.contrib.rnn.BasicRNNCell,\n",
        "    'memory_units' : 128,\n",
        "    'num_layers' : 1,\n",
        "    'l2_lambda' : 1e-3,\n",
        "    'dropout_keep_prob': 1.\n",
        "}\n",
        "\n",
        "\n",
        "# Specify the training hyperparameters:\n",
        "training_params = {\n",
        "    'num_epochs' : 100,   # Max epochs/iterations to train for.\n",
        "    'batch_size' : 100,  # Number of examples per batch, 100 default.\n",
        "    #'keep_prob' : 1.0,  # (1. - dropout) probability, none by default.\n",
        "    'train_only_on_fraction' : 1.,  # Fraction of training data to use, 1. for everything.\n",
        "    'optimizer_fn' : None,  # Optimizer, None for Adam.\n",
        "    'report_every' : 1, # Report training results every nr of epochs.\n",
        "    'eval_every' : 1,   # Evaluate on validation data every nr of epochs.\n",
        "   'stop_early' : True,  # Use early stopping or not.\n",
        "}\n",
        "\n",
        "# Build, train, evaluate and plot the results!\n",
        "trained_model, training_results = build_train_eval_and_plot(\n",
        "    model_params, \n",
        "    training_params, \n",
        "    verbose=True  # Modify as desired.\n",
        ")\n",
        "\n",
        "###############################END CODE TEMPLATE################################"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch: 0001 Training cost = 0.993639018\n",
            "Epoch: 0001 Validation acc= 0.957000017\n",
            "Epoch: 0002 Training cost = 0.509023278\n",
            "Epoch: 0002 Validation acc= 0.973399997\n",
            "Epoch: 0003 Training cost = 0.335925070\n",
            "Epoch: 0003 Validation acc= 0.980000019\n",
            "Epoch: 0004 Training cost = 0.230506954\n",
            "Epoch: 0004 Validation acc= 0.985400021\n",
            "Epoch: 0005 Training cost = 0.159965355\n",
            "Epoch: 0005 Validation acc= 0.984200001\n",
            "Epoch: 0006 Training cost = 0.116072138\n",
            "Epoch: 0006 Validation acc= 0.983600020\n",
            "Epoch: 0007 Training cost = 0.090510752\n",
            "Epoch: 0007 Validation acc= 0.985400021\n",
            "Epoch: 0008 Training cost = 0.070564545\n",
            "Epoch: 0008 Validation acc= 0.988799989\n",
            "Epoch: 0009 Training cost = 0.059962844\n",
            "Epoch: 0009 Validation acc= 0.987800002\n",
            "Epoch: 0010 Training cost = 0.051352463\n",
            "Epoch: 0010 Validation acc= 0.989199996\n",
            "Epoch: 0011 Training cost = 0.044365495\n",
            "Epoch: 0011 Validation acc= 0.986199975\n",
            "Validation loss stopped improving, stopping training early after 11 epochs!\n",
            "Optimization Finished!\n",
            "Accuracy on test set: 0.9844\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfgAAAC+CAYAAADZXh26AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8VPW9+P/XmX0mkz2TFQghgQQhAhIBURFFhIIUECjW\nutKv2nu/V2yV6ler9VoVa3svrXh/5coV5St1KS7oF8GlIouKEPbNELZAIHvIvkxmO78/JiSELQFy\nMsnwfj4eecxyts+8M2fen/M553w+iqqqKkIIIYQIKrpAF0AIIYQQnU8SvBBCCBGEJMELIYQQQUgS\nvBBCCBGEJMELIYQQQUgSvBBCCBGENE3wTz/9NKNHj2bKlCnnnefFF1/ktttuY+rUqeTk5GhZHCGE\nEOKKoWmCv+OOO1iyZMl5p69fv578/Hy++uor/vCHP/Dcc89pWRwhhBDiiqFpgs/KyiIsLOy809es\nWcO0adMAGDJkCLW1tZSXl2tZJCGEEOKKENBz8KWlpcTHx7e8jouLo6SkJIAlEkIIIYJDQBP8uXrJ\nVRSl3eU++uagFsURQgghgoYhkBuPi4ujuLi45XVxcTGxsbHtLrd01Y/YjDqyMtqfV1wahyOUsrLa\nQBcj6EmctScx1p7EWHsOR+hFL6P5EfyFxrIZN24cn3zyCQA7d+4kLCyMmJiYdtdpMuhY9lUutQ2u\nTiunEEIIEUw0PYJ//PHH2bx5M1VVVYwdO5ZHHnkEt9uNoijMnj2bm266ifXr1zN+/HisVisvv/xy\nh9Z7908G8ubKfbz79UEe/ukgLT+CEEII0SMpPXG4WK9P5fG/rONwYQ2P3JHJsAGOQBcp6EiTW9eQ\nOGtPYqw9ibH2umUTvRb0OoUHJg3EoNfx9pe51DW6A10kIYQQolvpkQkeIDEmhKk39KW63sX7a+Sq\neiGEEOJ0PTbBA0wc2Ye+8aFs3FvMrkPSQY4QQghxSo9O8HqdjjmTB6LXKbz9ZS4NTk+giySEEEJ0\nCz06wQP0ctiZcn1fKmub+Id0gCOEEEIAQZDgASaNSqZPrJ1vdxexN+9kQMpQV1fHihUfXvRyTzzx\na+rr6zQokRBCiCtZUCR4g761qX7p5/tpbOr6pvra2hpWrPjgrPd9Pt8Fl/vTn/5KSIhdq2IJIYS4\nQgW0q9rO1CculEmjklm58SgfrD3EvRMzunT7//3f/0VhYQFz5vwCvV6PyWQmNDSU/PxjvPvuRzz1\n1DzKykpxuZqYNevnTJniH0Vv1qyfsmTJMhoaGpg3by6ZmUPZu3cXDkccf/zjf2Iymbr0cwghhAgO\nQZPgAaZc35ftB8tYt7OQ7QfKMBr0nbbuazNi+dktaeed/i//8ghHjx7hzTffYceObTzxxG9Ytmx5\ny2h5Tz/9HKGhoTQ1NfHgg/dy0023NA+l2zq4zokTx3n++Zd58snf8fvfP8W6dd9w220TO+0zCCGE\nuHIERRP9KQa9jjmTBgJQ1+hGJXCd9F111aA2Q+EuX/4u999/Fw8//AClpaWcOJHfPKW1jAkJiaSm\n+isR6ekZFBcXdmWRhRBCBJGgOoIHSEkIY9KoZFZvOsawNAe/uG1AQMphsVhanu/YsY3t27eyePFS\nTCYTjzzyMC7X2QPlnN4cr9PpzzmPEEII0RFBdQR/ytQb+pIQbWPN9hPk5ld2yTZtNhsNDQ3A2SPo\n1dfXERoaislk4tixo+zbt/ec6+iBwwIIIYTopoIywRsNeuZMGoiiwFur99Pk9mq+zbCwcDIzh3Df\nfXeyaNFrbaaNHDkaj8fD3Xf/jNdf//8YPDjztKmt5+AVRUEIIYToDD1yNDmgQyMXLf/mEF9k53Pb\ntb25c1z/LihV8JDRobqGxFl7EmPtSYy1d8WMJtdR025MIS7Kxj+3HOfQiepAF0cIIYToMkGd4E1G\nPXMm+e+Hf3N1Dq4uaKoXQgghuoOgTvAA/XtFMC6rF8UVDXz6XV6giyOEEEJ0iaBP8AAzxqTiiLDw\nRXY+RwprAl0cIYQQQnNXRII3m/Q88JOBqKq/qd7tuXD/8EIIIURPp3mC37BhAxMnTmTChAksXrz4\nrOlFRUXce++9TJ8+nalTp7J+/XpNypGRHMnN1yRRWF7Pyo3SVC+EECK4aZrgfT4fL7zwAkuWLOGz\nzz5j1apVHD58uM08ixYtYtKkSaxYsYIFCxbw/PPPa1aeWWNTiQ6zsPqHfI4VB/aWjvHjxwR0+0II\nIYKbpgl+9+7dJCcnk5SUhNFoZPLkyaxZs6bNPIqiUFfnHw+9pqaGuLg4zcpjMRm4f1IGPlVlyaoc\nPN7ANdVLpzZCCCG0pGlf9CUlJSQkJLS8jouLY8+ePW3m+bd/+zfmzJnDsmXLcDqdvPXWW1oWiUF9\noxgzJJENuwpZ9cMxpt6Q0inrXbToNeLjE5g+fSYAb765GEVR2LlzB3V1tXg8Hh588FfccMNNnbI9\nIYQQ4kI0TfAd6SRv1apVzJgxg/vvv5+dO3fy29/+llWrVrW73IV69Vm28yM2Hd9+3um+CLANa+TL\n2vVs+d6MQd9+Q8ao3tdwz9AZ550+a9Z05s+fz0MPPQDAhg3fsGTJEv73/36YkJAQKisrmT17NtOn\n3w6Aolxaz0RdqbuXL1hInLUnMdaexLj70TTBx8fHU1jYOuRpSUkJsbGxbeb58MMPWbJkCQBDhw6l\nqamJiooKoqKiLrjuC3WL2NDowuu7cOUixGKkpsFFdb2LiBCTP+NeQEOj64LbjInpRWlpOfv351FZ\nWYnNZgcsvPjiy+zatROdTqGkpIQDB44RGRmFqnasu91Aka4nu4bEWXsSY+1JjLV3KRUoTRN8ZmYm\n+fn5FBQU4HA4WLVqFQsWLGgzT2JiIhs3bmT69OkcPnwYl8vVbnJvzx1pt3NH2u3tzrdk1Y98v6uY\nCWP6cfvovpe1TYCbbx7H2rVfc/LkScaNu42vvvqc6upq3nrrHXQ6HbNm/ZSmJhkCVgghhPY0vchO\nr9fz7LPPMmfOHG6//XYmT55MamoqCxcuZO3atQA8+eSTLF++nKlTpzJv3jxeeeUVLYvUxp3j+hNu\nN/H/vs+joKzustd3yy3jWbPmK9av/4abbx5HXV0dkZFR6HQ6tm/fSnFxUcu8PXSMHyGEED2Epkfw\nAGPGjGHMmLa3hM2dO7fleWpqKu+9957WxTinEIuR+yZksPCj3by5Ooen7xmOXnfpdZ6UlH40NDTg\ncMQRFRXNbbdN5MknH+O++35ORsZAkpNbL+iTq+iFEEJoKaiHi+2oxSv3sWlfCbPGpvKTUcmdtt6e\nTM6pdQ2Js/YkxtqTGGtPhou9RHfdOoCwEBMrvs2j6GR9oIsjhBBCXDZJ8IDdauSe2wbg8fp4c3UO\nvnauwBdCCCG6O0nwzYanx3JtRiyHC2r4euvxQBdHCCGEuCyS4E/zi9sGYLca+XjDEUoqGwJdHCGE\nEOKSSYI/TZjNxN23DcDl8fHW6v34eub1h0IIIYQk+DNdmxHLNQMcHDhexdrtBYEujhBCCHFJJMGf\nQVEU7rltACEWAx+uO0xZVWOgiySEEEJcNEnw5xBuN3PXrQNocntZ+vl+6XVOCCFEjyMJ/jxGDYpj\nSGo0OccqWb+zsP0FhBBCiG5EEvx5KIrCvRMzsJoNLF97iJPVzkAXSQghhOgwSfAXEBlq5s5xaThd\nXpZ+IU31Qggheg5J8O24ITOBwSlR7Mur4LvdRe0vIIQQQnQDkuDboSgK9/8kA4tJz/vfHKKytinQ\nRRJCCCHaJQm+A6LCLMy+JY3GJg//V5rqhRBC9ACS4DtozJBEruobye7DJ/lhX3GgiyOEEEJckCT4\nDlIUhfsnZmA26nnv64NU1UlTvRBCiO5LEvxFiImwMuvmVOqdHpZ9mStN9UIIIbotSfAXaeywJDL6\nRLDjYDmbc0oCXRwhhBDinDRP8Bs2bGDixIlMmDCBxYsXn3Oe1atXM3nyZKZMmcK8efO0LtJl0TVf\nVW8y6nj3nweprncFukhCCCHEWQxartzn8/HCCy+wdOlSYmNjmTlzJuPGjSM1NbVlnmPHjvHGG2/w\nj3/8A7vdTkVFhZZF6hSxkTZmjEnlvTUHeeerXP51emagiySEEEK0oekR/O7du0lOTiYpKQmj0cjk\nyZNZs2ZNm3mWL1/OXXfdhd1uByAqKqrd9S784U0qnVWalLmjxmX1Iq1XOFtzy9i6vzSgZRFCCCHO\npGmCLykpISEhoeV1XFwcpaVtk+HRo0fJy8vj5z//OXfeeSfffvttu+v9Ln8LL27+T74v2BywC910\nisKcSQMxGnQs+yqX2gZpqhdCCNF9aNpE35Hk6/V6yc/P55133qGwsJBf/OIXrFq1quWI/lx+de09\nvL3zQ97N/Yg9Vft4+Nq7iQ2J7syid4jDEcrdEwfy1mf7+OjbPH57d1aXl0FLDkdooItwRZA4a09i\nrD2JcfejaYKPj4+nsLB1qNWSkhJiY2PbzBMXF8ewYcPQ6XT06tWLlJQUjh49yuDBg8+73lv6jaa3\nqQ/v7f+IPSX7eezzPzAtdRI3Jo1Cp3TtjQHXXxXL+u3H2bCjgCEpUQwb4OjS7WvF4QilrKw20MUI\nehJn7UmMtScx1t6lVKA0zYaZmZnk5+dTUFCAy+Vi1apVjBs3rs08t956K5s2bQKgoqKCY8eO0bt3\n73bXHWEO51dXP8C9A2djUPQsP/AJC3cspqzhpCaf5Xx0OoUHJg3EoFd4+8tc6hrdXbp9IYQQ4lw0\nTfB6vZ5nn32WOXPmcPvttzN58mRSU1NZuHAha9euBeDGG28kIiKCyZMnc//99/PEE08QHh7eofUr\nisLIhOE8M/JxhsQM4mDVEeZnL2Dt8e/wqT4tP1obSTEhTL0hhep6F++vOdhl2xVCCCHOR1F7aHds\nZzYHqarKttJdLD/wCfXuBvqF9+XugbOIs3VNk7nX5+PFt7dxrLiWR2dezZC0mC7Zrlakya1rSJy1\nJzHWnsRYe92uib4rKYpCVtxQnh05j2GxV3Ok+igvZ/+Fr/PXd8nRvF6n45eTBqLX+ZvqG5wezbcp\nhBBCnE/QJPhTQk12/tfgu/lfg+/Borew4tAq/nPb3yiq175b2V6xdqaM7ktlbRP/+Eaa6oUQQgRO\nhxL86tWrqaurA+DVV1/ll7/8JXv37tW0YJdrWGwmz4x8nKy4oRytyeeP2X/ly6Pf4PV5Nd3upOuS\n6R1r59vdRezN69oL/oQQQohTOpTgFy1ahN1uZ/fu3Xz33XdMmzaNF198UeuyXTa7KYQHBt3Fw5n3\nYTPa+H9HvuA/tv0XBXVFmm3ToNcxp7mp/v9+vp/GJmmqF0II0fU6lOANBv/t8t9//z2zZs1iypQp\nNDX1nPHQr3YM4tmRjzMyfjj5tQW8smUhq/P+qdnRfHJ8KD8ZlczJmiY+WHdYk20IIYQQF9KhBK8o\nCqtXr2b16tVcd911ALjdPet+b5vRxr1XzeZfrn6AUJOdVXn/5JWtCzleW6DJ9qaM7ktSTAjrdhSQ\nc7T7D6AjhBAiuHQowT/zzDN89tlnzJw5k969e3P06FFGjhypddk0MThmIM+MfIzRCSMoqCviT1tf\nY+WRL3H7Orcp3WjQMWfyQBQF3vp8P06XNNULIYToOkFzH/ylyDl5gHf2f0hlUxUJIXHcM/BnJIe1\n34vexfhg3SE+35RPSkIoP7s5jfQ+kZ26fq3Ifa1dQ+KsPYmx9iTG2tPsPvg//vGP1NbW4vF4uOuu\nuxg6dCiffvrpRW+suxkYPYBnRj7GjUnXUVRfwp+3/hefHFqN29t5px+m3ZDCiIGx5BXV8sq7O3j1\ng12cKKvrtPULIYQQ59KhBL9x40ZCQ0P57rvviIuL48svv+TNN9/UumxdwmKwcGf6dB4d9hBRlkj+\nmb+Ol7e8ypHqY52yfqNBz6+mDuZ39wxnQO8Idh0+yXNvZvPmqhwqapydsg0hhBDiTBfV0c2WLVsY\nP348cXFxKIqiVZkCYkBkGr8b+Rhje11PSUMpC7b9jY8OrsTl7Zxx3lOTwnnyrmHMnXk1idEhfLen\niKcWb+KDtYeod/asCxaFEEJ0fx1K8NHR0Tz33HN8/vnnXH/99Xg8HrxebTuMCQSz3sSsAVP5zTX/\nQow1im+Of8v87L9wqCqvU9avKApD02J4fs4I5kwaiN1q5PPN+fyf//6BLzbn4/YEX0yFEEIEhv7f\n//3f/729mcaMGUNpaSn33Xcfffv2pbCwkNjYWAYOHNgFRTy3hobOObI+lyhLJKMTR+Dxedh3MpdN\nRVupczeQFtEPg05/2etXFIU+caHcPCwJq8XAgePV7DxUzsa9xYRYjPRy2APeQhISYtY0xsJP4qw9\nibH2JMbaCwkxX/QyHb6K3uPxkJeXh6Io9O3bt6Xzm0Dpqis2j1Qf4+85H1DSUEq0JYq7B85kQGRa\np26jrtHN6k3H+HrrCTxeH70cIcwcm0pmv+iAJXq5KrZrSJy1JzHWnsRYe5dyFX2HEvyePXuYO3cu\nJpMJVVXxeDy89tprDBo06JIK2hm68svk9rpZlfdPvs5fj4rKDUmjmJ46CYvB0qnbOVnt5JPvjrBx\nTzEqkN47glk3p9EvMaxTt9MRssN2DYmz9iTG2pMYa0+zBH/nnXfy6KOPtvRit2nTJv7617/y/vvv\nX3wpO0kgvkzHao6zLGc5RfUlRJoj+EXGTAZGD+j07ZworePD9YfZfdg/WE1WuoMZN6USF2Xr9G2d\nj+ywXUPirD2JsfYkxtrT7D74xsbGluQOMGrUKBobGy96Yz1dclhvnrz2USb2HUe1q4b/2vUG7+R8\nQKOnc2PRK9bOr2cN4cm7hpGSEMbW3DKeeWMzy77Mpbqu54wBIIQQInA6lOCtViubNm1qeZ2dnY3V\natWsUN2ZUWdgSr8JPJH1CEn2BDYWbeHFzQvYW57T6dtK7xPJM/cO51+nDSYm3MLaHQX8n9c38cm3\nR2SUOiGEEBfUoSb63bt38+ijj2IymQD/QDMLFy5k8ODBmhfwfLpDc5DH5+GrY2v54ug3eFUvI+OH\nM7P/FGzGzm9K93h9fLu7iE+/y6Om3kWozchPr0/hpqGJGPQX1Z1Bh0iTW9eQOGtPYqw9ibH2NDsH\nD/6knpeXh6qq9OvXD6PR2KENbNiwgfnz56OqKjNmzOChhx4653xffPEFv/71r/noo486dPFed/oy\nFdQV8fec5eTXFhBmCuXO9DsY4tDmAkSny8NXW47z+eZ8mlxeYiOs3HFTP7IyYtF14hX3ssN2DYmz\n9iTG2pMYa6/TE3x759nba6b3+XxMmDCBpUuXEhsby8yZM1mwYAGpqalt5quvr+fhhx/G4/Hw7LPP\n9rgED+D1efk6fz2r8/6JR/WSFTeUWf2nYjeFaLK9mnoXKzceZd2OArw+lb7xocwam8rAvlGdsn7Z\nYbuGxFl7EmPtSYy1dykJ/oI3sw8bNgxFUThVBzh1T7aqqiiKQk7Ohc877969m+TkZJKSkgCYPHky\na9asOSvBv/rqqzz44IO88cYbF/0Bugu9Ts+EvrdwtWMQf8/5gK0lO8mtOMTs9OkMi83s9O2FhZj4\nxfgBjM/qxccbjpCdU8qf39/J4H5RzLwplT5xF/9lEEIIETwumOD3799/WSsvKSkhISGh5XVcXBx7\n9uxpM09OTg7FxcXcdNNNPTrBn5IQEsfjw/+Vb45/y2dHvuSNvcsY5shketrtRFs7f6jY2Egbv5o6\nmAkjavhw3WH2Hqlg35EKRg2KY/qN/YiJuDIvhhRCiCudpt3RtXd6X1VV5s+fzyuvvNLhZU65lOaK\nrvTz2Nu5acC1LMpexo6yPews28uQ+IHc0u96shKvxqDv3NA7HKFcm5nIjtwylq7axw/7Stiyv4zb\nb0hh1rgBhIWYLmmdQnsSZ+1JjLUnMe5+OnyR3aXYuXMnr732GkuWLAFg8eLFAC0X2tXV1TF+/Hhs\nNhuqqlJeXk5ERASLFi1q9zx8Tznf41N9bCnewbcFm8ir8Q9BG2q0MzJhOKMTriUuJFaDbaps/rGE\nFRuOUF7txGo2MGlUH27N6o3Z2LG+9OWcWteQOGtPYqw9ibH2NL2K/lJ4vV4mTpzI0qVLcTgczJo1\n65wX2Z1yzz338NRTT3HVVVe1u+6e+GUqrCtmY1E22UXbqfc0AJAWkcLohBEMi70ak75jdyZ0lNvj\nY+2OAj7beJS6RjeRoWam3pDC9Znx6HUXvrVOdtiuIXHWnsRYexJj7V1Kgu/QaHKXSqfT0bdvX+bN\nm8e7777LtGnTGD9+PAsXLqShoYGUlJQ283/yySdcf/31xMa2f1TbE0cuCjXZuSo6nbG9byAxJI4G\nTyMHq46wq3wfGwo2UumsJtwcRri5c5q69DqF1KRwbhqahKLA/mOVbDtQxtb9pUSFmomPsp13MBsZ\nHaprSJy1JzHWnsRYe5qOJtfdBEttsbzxJBsLt7CpaAvVLv9n6hPai9GJI8iKG4q1Ewe0qaxt4tPv\n8vh2dyGqCmm9wvnZ2DTSeoWfNa/UyLuGxFl7EmPtSYy11+2a6LUUbF8mr8/LjxW5fF+4mb3l+1FR\nMemMXBM3hOsTR5ASltxpQ8cWltfz0frD7DhYDsCw/jHMHJtKQnTrPfuyw3YNibP2JMbakxhrTxJ8\nkKhqqmZT0TY2FmZz0lkBQHxIHNcnXMuI+OGd1nnOwRNVfLD2MIcKqlEUuPHqRKbekEJkqFl22C4i\ncdaexFh7EmPtSYIPMj7Vx4HKw2wszGZX2V48qheDomeIYzCjE0cwIDIVnXJ5/dCrqsrOQ+V8uO4w\nRScbMBl0jL+2N/dMHkRDnbOTPok4H/lh1J7EWHsSY+1Jgg9ida56sku2831hNsX1JQBEW6IYnXgt\noxKyiDCffR79Ynh9Pr7fU8wn3x6hqs5FiNXI8AExZKXHkpEcqcmANkJ+GLuCxFh7EmPtSYK/Aqiq\nSl5NPhsLs9lWshOXz42CwqDoDK5PHMGg6Az0uo7d634uTW4vX289zjfbC6is9Y89H2IxMKy/g6wM\nB1f1jZJk34nkh1F7EmPtSYy1Jwn+CtPocbKtZCffF2aTX3sCgHBTKCMTshidMAKHLfqS1x0dbeeH\nnSfYur+UbQfKWpK91WxgWH//kf2glCiMBkn2l0N+GLUnMdaexFh7kuCvYMdrC/mhKJvs4h00evyj\nAA6ITOP6xBEMiRmE8SI70Tl9h/WpKkcKa9i6v5StuaVU1PiTvcWkZ2hzsh+cEoWpg73kiVbyw6g9\nibH2JMbakwQvcHnd7Czbw/eFmzlUlQdAiMHGiPhrGJ04gkR7fIfWc74dVlVV8opqW5J9ebX/Qjyz\nSc+Q1Giy0mPJTI3ucJe4Vzr5YdSexFh7EmPtSYIXbZQ0lPFD4RY2FW2l1l0HQEpYH0YnjuCa2CFY\nDOfvGakjO6yqqhwrqWXL/lK27i+lrMqf7E1GHVenxpCV7uDq1GgsJk3HNOrR5IdRexJj7UmMtScJ\nXpyTx+dhb3kO3xdlk3PyACoqZr2JrLihjE4cQXJo77M60bnYHVZVVY6X1rUk+5JK/2kCk0FHZr9o\nhmc4GJIag9Usyf508sOoPYmx9iTG2pMEL9pV4azkh6Kt/FC4hcqmKgCS7AmMThjBtfHDCDHagMvb\nYVVVpaCs3p/sc0spOukfWMeg15HZL4qs9FiGpMVgs0iylx9G7UmMtScx1p4keNFhPtVHTsVBNhZm\ns7t8Hz7Vh0FnYJgjk9GJIxjdfwjl5XWdsq2Csjq25paxNbeUgrJ6AAx6hUF9o8jKiGVo/xhCLJ07\nkl5PIT+M2pMYa09irD1J8OKS1Lhq2Vy0jY1F2ZQ2+PunjwuJIT1iABlRafSPSMVmtHbKtopO1jdf\noFfG8VJ/BUKvU7iqbxRZ6Q6GDXBgt145yV5+GLUnMdaexFh7kuDFZVFVlUNVeWwsymZX+T6aPP7b\n4RQU+oT1IiOyP+mRafQLT77o2+7OpbiigW25pWzdX8axEv//U69TyEiOJCvdwTUDHITaTJe9ne5M\nfhi1JzHWnsRYe5LgRaeJjLKy5ciP5FYcJLfyEHk1+fhUHwBGnYHU8BTSo9LIiOxPr9DEy+4Tv7Sy\ngW3Nzfh5Rf7/rU5RSO8TQVZGLNcMcBAeEnzJXn4YtScx1p7EWHuS4EWnOXOHdXqcHKrKI7fyEPsr\nDlJYX9wyzWawMiAyjfTINDKi0nBYYy5raNvyqka25paxLbeUw4U1ACgKpPeOYHh6LMPTHUTYz3+L\nX08iP4zakxhrT2KsPUnwotO0t8PWuGo5UHGI3MpD5FQcbLkiHyDSHNFydD8gMo1w88V/MU85We1k\n2wH/kf2hE9UAKED/XuEMz4glKz2WyNCem+zlh1F7EmPtSYy1JwledJqL2WFVVaWs8SS5lYfIrTjI\ngcrD1HsaWqYnhsSTHuU/wu8f0Q+LwXJJZaqsbfKfs88t4+DxKk59cdOSwslKdzAkLYbYSOtltR50\nNflh1J7EWHsSY+1Jghed5nJ2WJ/q40RdIbnNR/iHqvJw+9wA6BQdfcN6k958wV5KeB8Muou/H76q\nrontB8rYur+U3ONVnPoW261GUhPD6JcUTmpiGCkJYd26cx35YdSexFh7EmPtdcsEv2HDBubPn4+q\nqsyYMYOHHnqozfSlS5fywQcfYDAYiIqKYv78+SQkJLS7Xvkyaaszd1i3z0Ne9TFyKw6yv/IQx2qO\nozYff5t0RtIi+/nP30f2J9Eef9EX7FXXu9hxoIz9+ZUcLqjhZI2zZZoCJDlCSE0Kp19iGGlJ4cRF\n2dB1k6N8+WHUnsRYexJj7XW7BO/z+ZgwYQJLly4lNjaWmTNnsmDBAlJTU1vmyc7OZsiQIZjNZt57\n7z2ys7P5y1/+0u665cukLS132AZ3IwerjrQ06Rc3lLZMsxtDSI9Ma27S70+MNeqi119V18SRwhoO\nF1RzuLCGo0U1uDy+luk2s4F+iWGkNh/l90sMwxagjnbkh1F7EmPtSYy1dykJXtO2y927d5OcnExS\nUhIAkydPZs2aNW0S/IgRI1qeDx06lJUrV2pZJNEN2IxWhjgGMcQxCICqpuqW5vzcykNsK93FttJd\nAMRYolriDzGVAAATdklEQVSSfXpkGnZTSLvrj7CbuWaA/z56AI/XR0FZPYcLq1uS/t68CvbmVbQs\nkxBtIzUxnNSkMFITw0mMCUGn6x5H+UIIcSk0TfAlJSVtmtvj4uLYs2fPeef/8MMPGTNmjJZFEt1Q\nhDmckQnDGZkwHFVVKWkoa71gr+ow3xdm831hNgC97IktCT8tIgWzvv174w16HcnxoSTHh3LLNb0A\nqGlwtRzlHyms4UhRDUV7ivhuTxHgH+s+JaHtUX6wd7ojhAgumib4i2n9//TTT9m3bx/Lli3r0PyX\n0lwhLk6gYhxLGJl9U4EJeH1e8iqPs7skh70luewvP8yJukLW5G9Ar9MzILofmXEZZMalkxqZjEHf\nsa+0A0hNjmZ882uvTyW/uIbcY5XsP1ZB7rFKcpr/TkmMCSE9OZKMvlGk94mkb0IYev3ldfAD8l3u\nChJj7UmMux9NE3x8fDyFhYUtr0tKSoiNjT1rvo0bN7J48WL+/ve/YzR27FyonO/RVnc6pxZONDc6\nbuBGxw24vG6OVB9lf8VBcisPsr/sEDllB1m+dyUKChHmcKKtkURbooi2RBJtbX2MMIdf8AI+u1HH\n8LRohqdFA1DX6CavqPVc/pHCGtZuO8HabScA/7j3KfFh9EsKIy0xnH5J4Rfd2153inOwkhhrT2Ks\nvW53Dj4zM5P8/HwKCgpwOBysWrWKBQsWtJnnxx9/5LnnnmPJkiVERkZqWRwRBEx6IxlR/cmI6g9A\nvbuBA5WHya08RGFdMRXOSg5XHeUQeWctq1N0RJkjiLJGEWOJJMoS1VoZsEYSZgptUwGwW41k9osm\ns58/4ftUlaKTDRwpqPafzy+s4cDxKnKPt3byExNuaXPFfu9YO4ZOOMoXQoiL1SW3yb300kuoqsrM\nmTN56KGHWLhwIZmZmdx888088MADHDx4EIfDgaqqJCYm8re//a3d9UptUVs9uUbu8XmodFZz0lnh\n/2usbHmscFZQ7Tr35zLoDERZIlqP/psTf5QlihhrFHZjyFmd6DQ2eThSVNOc9P1H+/VOT8t0o8F/\n/j81Maz5Ir7wNj3v9eQ49xQSY+1JjLXX7W6T05J8mbQVzDusy+umwlnJSWclJxsrqHBWUu6soKK5\nIlDnrj/nciadkShr2+R/+qkAm8E/pG5JZWNrs35BNcfL6jh9L4sMNfuP8hPCGJgag1nnP/KXI31t\nBPN3ubuQGGtPErzoNFfyDuv0NDVXAE47+ndWUtFYQbmzkkZP4zmXs+gtZyX9KEskoYYw6qqMHC92\ncrighiOF1dQ0uNssqygQHWbBEWElNtJKbIS15bkjwtqte+Pr7q7k73JXkRhrr9udgxeiJ7IYzCTa\n40m0x59zeoO70Z/wnRWcbPQn/1OVgbLGkxTUFZ1zuRCDjajkSAalR2JVwvA5LSi+EKrKoLpaT3mF\n96wr908JtRnPSvyxETYckVbCbMYe1f++EKJrSIIX4iLZjFZsRiu9QxPPmqaqKvXuhpaj/jMrAMX1\nJRyvLWi7kNX/Z0w00scchk0XikkNAbcVj9NMY52BmkoXR0sbOVygx98BbyuzSY8jvDnpn6oEND9G\nhZnR66TpX4grkSR4ITqRoijYTSHYTSEkh/U+a7qqqtS46lqO/l0GJycqSqh0VlPZVEWls4oy98nW\nBfRAuP/P1BdMOhMhen8FQPFY8TSaaawzUlqtoyDfjHrIAr7W3VqvU4gOtxAbcXbyd0RYMRn1msdE\nCBEYkuCF6EKKohBuDiXcHEpKePI5z126vW6qmmpaEn5lkz/5V5167qyi0tNcCWg++tc7/HUBAKNi\nxtxcAfA6zdTXG8mpMfFjuQXV5f9D9c8dYTe1Jv1IW0tFwBFhxW4NTP/8QojOIQleiG7GqDfisEXj\nsEWfd54mr4tKZxVVpxJ+U1VrK0BTNVXOKpxUgB2wgymu7fIG1YLO4z8FkNdg4ki5BbXQgtrUXAlw\nW7CZTC1H+6eSviPcQqjNRKjNSIjVKFf+C9GNSYIXogcy603Eh8QSH3J2z5CnNHoa2zT9+ysDpyoB\n/gqB11iJ4VwX56qA10Jxk5miJgtqgQU1rzn5e4yoHiN4jFj0VuwWK2E2E6FWE3abkVCrsfmx+XXz\ne6E2ExaTXi4IFKKLSIIXIkhZDVasdut57wZQVZV6TwOVzmqqTj8dcNrrKmM1npDqC26nToVajxHV\n60/6aqURtaz5ube1MqB6jOhVE1aDDbsxhDCLjTCbGXtz8vc/tlYG7DYjdmklEOKSSYIX4gqlKAp2\nYwh2Y8g57wgA8Kk+6tz1Lcm/uqmGBncjDZ4G6t0NNHgaaXD7n9e7G6j31OJTfRfcrhuobP5TvXp/\nBaDShFrWXBk4o1JgUszNlQIrdnMIERY7YTYroc0VgFCr/5TBqVYDq1laCYQASfBCiAvQKTrCTKGE\nmUJJ5uy7As6kqioun7sl6bdUADwNNLgbm99roL75eZ2rnnp3I42eRtxqzXnX29j8V3ZqO14dNLcU\nnF4pUD1GFJ8Rs86CVW8lxGgjwhaKUTEQYrJiN1sJNVuxW03YzAZsFgM2swFr83OzUSoHInhIghdC\ndBpFUTDrTZj1JiItERe1rMfnOa1FwN9K0OBubK4c+CsMNU311DbVt0x3ep241Lqz1uUF6pr/Sk6f\n0OT/Uyt14DOgevXgNaD6/I/4DBgwYlBMGHUmzDoTZoMZi96MzWTGZrRiN1mxmy2EWmyEWayE22yE\nWsxSQRDdjiR4IUS3YNAZWloLLoZP9eH0OFuS/uktB7WuejyKh/KaGhrdTTR6nDg9TTR5m3D73LhV\nFx7ViRcPKP5eu1X8pxHcQMOZGzs14YzhClSfDporC4pqQIextaKgmDDpTVj0ZiwGM1ajmRCjhRCT\nrbmiYCXcavNfk2C1YjFYMChSURCXTxK8EKJH0yk6bEYbNqMNOPvWwo70k66qKm6fG6e3qaUC0OR1\nUe9qpNbZSG1TA/UuJ/VNThrcztMqCi5cviY8qhuP4sJrcONTnPh0tbgUcJ25IW/zn7MDH0xVQFVQ\n0DX/tT7XoUNRdOjQo1N06BVd86MevaJDr9P7n+v0GHQ6DDqD/1Gvx6AzYNTrMer0GPT+5/75W5fX\nKXr0utPX2bwd3WnTm7er1+mp1oVSU+tsne+s5U+tU99SVqE9SfBCiCueoiiY9P4j7YttQTgXf4XB\nQ72rkarGemoaG6lxNlDrbKTO1Ui9q7mi4G6iyetsrii4cPtceFQ3Pryo+FDx4UNFxedvYVBUULwo\niue0163TFKVnjB2moKA7vVKiO7MicXol4uwKQptKhq61UnG+ykXLuk/bzuCYDKIskYEOhaYkwQsh\nRCfzVxiMmKxGIq1hl70+VVXxeFXcHi9ujw+Xx4f7tD/XqffdXpo8Hpo8blweDy6vF5fbg8vraXnt\n9nhw+zz+dfm8eLwe/6PPi8frf/T6vHhU/3NoW5FQzqhUtH3/7Gmt759ZGTltPlQUnRcUj39+ndq8\nXV/b5TrRvuKhTEy6Hb1OwaDXodcrGHTNj3pd8/sKer0OXQ89XSIJXgghujlFUTAaFIyGrm3aVlUV\nr089qyLR+tr/aAsxcbKiAY/Xh9vrw+Px+Sskp577fHg8auv0c83j9b8+cx5383sejxcVLlC5OP39\nMysXZywDbKmNZItna4fioChg0Ov8Cf9UJUDnT/5tKgM6XUuloKXicFpF4dQyp6b5l2mdT3/aNs5c\nZpwMFyuEEKKzKIo/ORn0Oqzm88/XFePBn6psnF4R8FcATqsYtFQUfLg9Kl7f6e+pbaZ7vP7p/kf/\n8l6vr/m5itfrw+Nr++j1qq3veVWa3F4anB68vuZpXhWfqs1pknGj+l70MpLghRBCdHunVza6M5+q\nNid7f2WhtdJwRmXitIrCmRWN1umtFYxLIQleCCGE6CQ6RUEXgNMp5yyL1hvYsGEDEydOZMKECSxe\nvPis6S6Xi9/85jfcdtttzJ49m8LCQq2LJIQQQgQ9TRO8z+fjhRdeYMmSJXz22WesWrWKw4cPt5nn\nww8/JDw8nK+++or77ruPP//5z1oWSQghhLgiaJrgd+/eTXJyMklJSRiNRiZPnsyaNWvazLNmzRqm\nT58OwIQJE/jhhx+0LJIQQghxRdA0wZeUlJCQkNDyOi4ujtLS0jbzlJaWEh/vH85Sr9cTFhZGVVWV\nlsUSQgghgp6mF9mpHbhd4Mx5VFXtUB/Mjku4J1BcHIlx15A4a09irD2Jcfej6RF8fHx8m4vmSkpK\niI2NPWue4uJiALxeL3V1dYSHh2tZLCGEECLoaZrgMzMzyc/Pp6CgAJfLxapVqxg3blybeW6++WZW\nrFgBwBdffMGoUaO0LJIQQghxRVDUjrSjX4YNGzbw0ksvoaoqM2fO5KGHHmLhwoVkZmZy880343K5\n+O1vf0tOTg4REREsWLCAXr16aVkkIYQQIuhpnuCFEEII0fUC39WOEEIIITqdJHghhBAiCEmCF0II\nIYJQj0vw7fVtLy5PcXEx9957L5MmTWLKlCm8/fbbgS5S0PL5fEyfPp1f/epXgS5KUKqtrWXu3Ln8\n5Cc/YfLkyezatSvQRQpKS5cu5fbbb2fKlCk8/vjjuFyuQBepx3v66acZPXo0U6ZMaXmvurqaOXPm\nMGHCBH75y19SW9v+8Lw9KsF3pG97cXn0ej1PPfUUq1ev5v333+edd96RGGvk7bffJjU1NdDFCFov\nvfQSN910E59//jmffvqpxFoDJSUlLFu2jI8//piVK1fi9XpZvXp1oIvV491xxx0sWbKkzXuLFy/m\nuuuu48svv2TkyJG8/vrr7a6nRyX4jvRtLy6Pw+Fg4MCBAISEhJCamnpW98Li8hUXF7N+/XpmzZoV\n6KIEpbq6OrZu3cqMGTMAMBgM2O32AJcqOPl8PhobG/F4PDidzrM6MxMXLysri7CwsDbvnT5uy/Tp\n0/n666/bXU+PSvAd6dtedJ4TJ06wf/9+rr766kAXJejMnz+fJ554okPdMouLd+LECSIjI3nqqaeY\nPn06zz77LE6nM9DFCjpxcXE88MADjB07ljFjxhAaGsro0aMDXaygVFFRQUxMDOA/EKusrGx3mR6V\n4OWW/a5TX1/P3LlzefrppwkJCQl0cYLKunXriImJYeDAgfKd1ojH4+HHH3/krrvuYsWKFVgsFrlm\nRwM1NTWsWbOGtWvX8u2339LQ0MDKlSsDXSzRrEcl+I70bS8un8fjYe7cuUydOpVbb7010MUJOtu3\nb+ebb75h3LhxPP7442zevJknnngi0MUKKvHx8cTHx5OZmQn4h6L+8ccfA1yq4LNx40Z69+5NREQE\ner2e8ePHs2PHjkAXKyhFR0dTXl4OQFlZGVFRUe0u06MSfEf6theX7+mnnyYtLY377rsv0EUJSo89\n9hjr1q1jzZo1LFiwgJEjR/KnP/0p0MUKKjExMSQkJJCXlwfApk2b5CI7DSQmJrJr1y6amppQVVXi\n3InObN275ZZb+PjjjwFYsWJFh3KfpsPFdja9Xs+zzz7LnDlzWvq2ly9T59q2bRsrV65kwIABTJs2\nDUVR+M1vfsOYMWMCXTQhLsozzzzDvHnz8Hg89O7dm5dffjnQRQo6V199NRMmTGDatGkYDAauuuoq\nfvaznwW6WD3eqZa9qqoqxo4dyyOPPMJDDz3Eo48+ykcffURiYiKvvvpqu+uRvuiFEEKIINSjmuiF\nEEII0TGS4IUQQoggJAleCCGECEKS4IUQQoggJAleCCGECEKS4IUQQoggJAleCNHpsrOzWwZ6EUIE\nhiR4IYQmZCAdIQKrR/VkJ4S4fLt37+Y//uM/qK+vB2Du3LmkpaUxY8YMpk+fzvfffw/A73//e7Ky\nsgD45JNPWLJkCTqdjj59+vD888+39IX9+uuv89lnn6HT6bDZbLz33nuAf0yD3//+9+zcuROdTseC\nBQvo169fAD6xEFcoVQhxxaipqVGnTZumlpWVqaqqqqWlpeqYMWPUnJwcNT09Xf30009VVVXV7Oxs\ndcyYMarL5VJzc3PVG264QS0vL1dVVVX/+te/qr/+9a9VVVXVjz/+WJ09e7ZaX1+vqqqqVlVVqaqq\nqps3b1YHDRqk5uTkqKqqqosWLVLnzZvXpZ9ViCudHMELcQXZvn07J06c4MEHH2wZzEKv1+PxeDCZ\nTPz0pz8F4Nprr8VisZCXl0d2djZjx44lOjoagDvvvJOpU6cC/qFvf/7zn2Oz2QAIDw9v2VZKSgoZ\nGRkADBkyhHXr1nXVxxRCIE30QlxxMjIyWLZsWZv3CgoKzppPVVUURWl5PPP9U8/Px2w2tzw/VYkQ\nQnQduchOiCvIsGHDOHr0KJs3b255b8+ePaiqisvlYuXKlQBs3boVl8tFSkoK1113HevXr+fkyZMA\nLF++nNGjRwP+ISzfe++9lvP5VVVVXfyJhBDnI0fwQlxBwsLCWLRoEa+88govv/wyLpeLPn368Lvf\n/Y6IiAhycnL4n//5HwAWLFiAwWAgLS2Nxx57jPvvvx+dTkfv3r35wx/+AMC0adMoLS1l9uzZ6PV6\n7HY777zzTiA/ohCimQwXK4SgoKCAGTNmsGnTpkAXRQjRSaSJXggByH3rQgQbOYIXQgghgpAcwQsh\nhBBBSBK8EEIIEYQkwQshhBBBSBK8EEIIEYQkwQshhBBB6P8HGwE9P8REAA8AAAAASUVORK5CYII=\n",
            "text/plain": [
              "<matplotlib.figure.Figure at 0x563cf0d49e90>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAf8AAAC+CAYAAAA7ggbDAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xt8FOW9+PHPzOxu7hcSspsQIUCoRiAC/YFcRTDVUC5C\nTCqoPxHp0VqL1KqFioK/FqsFldcR9FA4pXL02J72KGgh2qYNAlYkiAIBAWtTIJJAEsg9m+xmd+b3\nx4Yl4ZYlMElIvu/XK6+5PTPznUfMN88zl0cxDMNACCGEEN2G2tEBCCGEEKJ9SfIXQgghuhlJ/kII\nIUQ3I8lfCCGE6GYk+QshhBDdjCR/IYQQopsxNfkvWrSIMWPGMG3atIuWef7557njjjuYPn06hw4d\n8q/fuHEj6enppKen895775kZphBCCNGtmJr877rrLtatW3fR7du2baOwsJCcnBx+8Ytf8NxzzwFQ\nVVXF66+/zjvvvMP//u//8tprr1FTU2NmqEIIIUS3YWryHz58OJGRkRfdnpuby4wZMwAYMmQINTU1\nnDp1ir///e+MHTuWiIgIIiMjGTt2LB9//LGZoQohhBDdRofe8y8tLSU+Pt6/HB8fT0lJCSUlJSQk\nJPjXOxwOSkpKOiJEIYQQosvp0OR/7peFDcNAUZTz1gMoinLZxxNCCCHE+SwdeXKHw8HJkyf9yydP\nnsRutxMfH09eXl6L9aNGjWr1eIqiUFYmzwaYLS4uQurZZFLH5pM6bh9Sz+aLi4u47H1Mb/lfqjWe\nlpbmf5J/7969REZG0rNnT8aNG8eOHTuoqamhqqqKHTt2MG7cOLNDFUIIIboFU1v+Tz75JHl5eVRW\nVjJhwgQee+wxGhsbURSFmTNncuutt7Jt2zZuv/12QkJCePHFFwGIiori0UcfJTMzE0VRmDdv3iUf\nHBRCCCFE4JSuNqSvdC+ZT7rxzCd1bD6p4/Yh9Wy+tnT7d+g9fyGEEEK0ja4beHW9TftK8hdCCCEC\nYBgGHq9Oo8c39Xh1Gr06Hs+ZqeGbNlvX6Gla9hrN5n3rG5vWezxnj+Uv49Fp9Brnl/c07ePV8eq+\njvtNr0y/7GuR5C+EEKJT0vXzk+mZZNlyfcskecFk25Sc/fPNEunZpHr+OZofw+Nt37vkFk3FalGw\naCoWTSXIqhEebPUtWxSsmorF0rbn9iX5CyFEN2cYBl7daJE0vc0T4TmJ1ts8gZ7Z3pQsvd7mCdvA\nYtOoqXU1S8zNErA/4Z5t/TZP1Lqpj6QZoOig6qDoKKqOqhlYCMZKEBbNl3SDQ63+JGvRVKyagtWi\n+bb71zVNLSoWTWmanl13dn/FX9bSbL216Vxn1lktKpqqBPR9m7aS5C+EEO1MNwwaG73UuOupddVR\n43JS43ZS1+ikrrEeZ6OTek8D9Z566r31uPQGXN4GvIaOgoqCAoYKRtO8rmIYChgKhqFg6KpvXlfQ\ndTB0tWmqoHsVdF3B6wVDB49XwfCCYTTt0zTF8B235bqWU+MC6zAU4NykZYDSMtmqqo5mBYtmoFkM\ntBADiwVsmoGqGWiagarpKJqBqhooqo6iGiiqF1QDpdmxDMUL6BhN8zo6Bl50vOiGb+o1vHjx4jU8\neHVfmQsJsQSzbNxzaKpm9j+DDiXJvx3U1tby17/+mYyMrMvab8GCx3nuuecJCws3KTIhugfdMJoe\njjo7tVQ3cKqq3t8aPdO92/ze6plu3+bbmpdp9Hhp8PoSs8tw4dZduPUGPLhoxIUXN17FhVd1oytu\nDK0R1EawNILWyOU07Ay9KcEqOihGy32v4IstZiQBFRVV8f3oho7H8AIXb8V7m37cl3si48KHVVCw\nqBoW1YJFsWBRLQSpQb5l/zrt7HLTOquqkRAe3+UTP0jybxc1NdVs3Pi/5yV/XddR1Yv/X7t8+b+b\nHZoQAdMNA5fbS4PbS73LQ73bQ4PLN9/g9uLR9RYJ9txkqxvnLF9i+9lt+tkyF0jgl9737PTSnccG\naB4UzZeUFYvHl5gtjc3WNfrK+OebylkbUWyB16Giq6iGDU0PQfNEoxk2rARhVXw/NjWYoKafYEsw\nIVoIoZYQQizBBGk2bFYNq6aiaSqaaqBqNLWK8XVbqwaoTesUHVQD0NHR8epevIbuawFfbN7wol9i\nm1dvOa+fmb/ANq/hRTd0gmxW8KpY1XMS7kUS8LnrrMo5Zc4cRzln2Z/ALaiKamqXeVcgyb8d/PrX\nr1FcXMTcufehaRo2WxAREREUFh7jd797l6effoqyslLcbhff+949TJvmG+nwe9+7k3Xr3sLpdPLU\nU/NJTR3KgQP7iItz8KtfvYLNdhm/dbqwRt1DlauaSlcVVa4qKv3zvmldoxNVUbGoGlrTLxdN0XxT\n1YJF0dBUrWl67rLvl4ymXqpMs+ULneO8Za1dfzF5dd2fsBtcXurdHupdXhrcHl8S9897mxK6h3q3\n1z/1JXffvu37uFNTt67qRdGauoktvq5gX3ewjmrzTRVVR9G8WFTfPKrXP0XxYqh607RpuWne29Q6\nP6+X+hIsioUgLZgQLZwQLYQQqy9Bh1lDCbeFEG4LJcwaSqg1hFDLmanvx6pZzauuTkre8++culXy\n/+OWf/LZ4dKreswRKXbuvm3AJcv88IePcfTov/jtb99mz57PWbDgJ7z11h/9IxouWvQcERERuFwu\nHnpoNrfeelvTFw3P/kY6fvwbfv7zF1m48BmWLHmarVu3cMcdk67qtXQ2hmFQ53H6k3hlU2I/N8HX\nNtZd9BgKCmHWUH/Xo68l423Hq7gwVVEv/oeEqmGzWvB6jKZ7qpy9n6uDbjTduzVA94Kun/3xesHr\nNfDq4PWAx2ug65y9P4vS1FV6Zl5pOc/Zc6mKgtVmwRaqEmGxEGPRsGm+2GwWC0EWjSCrb2qzWkBt\ndp8VT9OPFy8e34/RbGp48Jz50T14jEb/tFH30Kg3ohsXf3/5TG9vW95w9rVArYRYgwhWYwixhJxN\n0P5EHXreupCmdVa1W/3aFF2U6f+Kt2/fzgsvvIBhGGRmZvLwww+32F5cXMyiRYsoLy8nOjqal156\nCYfDAcBLL73Etm3bMAyDMWPG8Mwzz5gdbrsYOHBQi6GM//jH3/Hxx9sA3zDHx48XMnDgYJrfzEpI\n6EVysu+PjBtuSOHkyeJ2jflq8/hb6+e32M8k+Cp3NY2656LHCNJsRAdF0Ss8geigSKKDoohqmp5Z\njrCGn3f/zjAMvIYXT9MfAr6p58LLurfpjwaP/4+Hc5d96zzNtvmmHt1Do8dDg6cRt9eDu2na6PUd\n26N78Xo9uA3f+XTDjYHvgSXjzANSTdNLdhSoXPSe7yU2BcQAXE0/F9TY9FPftuOriq872KpasapW\nwiyh/nmrZsV2ZpvmW2fzz1ualbGeU8bin7eqVmzN5i2qhqr4akRapKI7MzX567rO0qVLWb9+PXa7\nnaysLNLS0khOTvaXWbZsGRkZGUyfPp28vDxeeeUVli9fzp49e9izZw+bN2/GMAzuuecePvvsM0aM\nGNHmeO6+bUCrrfT2EBwc7J/fs+dzvvhiN2vXrsdms/HYYz/A7T7/sZfmXfyqql2wTGdgGAZOT/05\nrfTzW+yttdYjbeEkhMX7E3lUs4R+ZjnEEnzRY1yKoij++4WBcDd6qWvwUNfQiLOxadrgwd3gwdnQ\nSF29hzqXb92ZbXUNHurqG/0f4QiEpiqEBVsIDbYSFe571SjEphEcZCHYphFsUwmyaQQHKQTbNGw2\nlSCbQpBVw2ZTsFlVbFYFRQHd0H0PuRk6BkbTsm+dwdltuqFjGIbvrvCZ7YbvWekW2w29qUzT9mb7\nGIaBYRhYVO2iSdeqWS6QpK3d4sEqITojU5N/fn4+SUlJJCYmAjBlyhRyc3NbJP+CggIWLVoEwMiR\nI3n00UcB3y9ol8uFy+VC13U8Hg+xsbFmhmua0NBQnE4ncP4oh3V1tURERGCz2Th27Chffnnggsfo\nDEMw6IZOpauK02WlHC050abWuk2zER0USa+w+Asm9OigSCJtEVc9KTR6ziTwpoTdPHE3S9hntjVf\n5/EG3rmsKgqhwRbCQqz0jAr2zQdb/VNfcj8733ybzXr2ISVplQohzGRq8i8pKSEhIcG/7HA42L9/\nf4syKSkp5OTkcP/995OTk4PT6aSqqoqhQ4dy8803+4fyve++++jfv7+Z4ZomMjKK1NQhPPDALGy2\nIGJiYvzbRo4cw3vvvcv//b9306dPEoMHpzbb82xfb3s8IGYYBjWNtZyuL+d0fTmnGip88w2+5XJX\n5UXvwyooRNjCSQhzNCXxC7fYg7Xgq3YthmFQ1+ChstZFVa2bylpX04+bqqZpZa2L6jo3bk/gCVxR\n8CflmMggQi+QqH1Ty3nbgm3t+zCfEEK0hanJP5DW6oIFC1i6dCkbN25k+PDhOBwONE2jsLCQI0eO\n8PHHH2MYBg8++CC7d+9m+PDhlzxeW0Y3ag+rVl38tb3/+q83Lrh+69Yt/vkPPsj2z8+f/8M2x+F0\n11Nad4rSutO+ae1p/3JZ3Wlc3gvfTogKjiQ5Jgl7WCyxoTHEhEQRExLt+wmNJjo4CstVaq0bhkFt\nfSPlVQ2crm6gorqB8mY/FdUu//rGSyR1VVWIDg+id3wEESE2wkKtRITaCA+x+n5CrYSH2JqmVsKb\ntoUEWVDVjk/gnfXfclciddw+pJ47H1OTf3x8PMXFZx9MKykpwW63tyhjt9tZtWoVAE6nk5ycHMLD\nw/nDH/7AkCFD/PfHb7nlFvbt29dq8u/uXaWN3kZON1T4W+unGso5XX922em58JNZIZZg4kJ6EhsS\nQ2xwD2JDYugZHONftmlnnzlo0SVtgFEHFXXOVmNr3lKvrHVRWeOmqs43raxzNWvBuy/Z1a4qClHh\nNhJ7hhEdHkR0RBDRYTaiI4KICrP51oXbiAi1XV4S93px1npx1ga+i1mk2998UsftQ+rZfJ1uSN/U\n1FQKCwspKioiLi6O7OxsVqxY0aJMRUUF0dHRKIrCmjVryMzMBCAhIYF33nmHhx9+GF3X+eyzz5gz\nZ46Z4V4TdEOnoqHKn8xPN5Rzqr6C0w2nOV1fTpX7wv+TWVULMcEx9I3q0yypxxAb0oOewTGEWkPb\nHJM/qde4fEm8xt2yK77ubKK/1MAYZ5J6b7svqUc1JfEzyTwqzJfoI0KsnaJlLoQQ1ypTk7+maSxe\nvJi5c+diGAZZWVkkJyezcuVKUlNTmThxIrt27WLFihUoisKIESNYsmQJAJMmTWLnzp1MmzYNVVW5\n5ZZbmDBhgpnhdgptve+uKio9gqK4Pjq5ZWJvmo+whftfcWoLXTcoLK3h8LFK6txeTpbVnr2/3kpS\n11SFyDAbve0R/mQe1Sypn0n0EaFWVLlfLoQQplOMzvAY+VV0rXQv6YbOv6qOcaz6m2Zd9BWU15fj\n1hsvuE+kLaJFa/1sko+hR1DUVX9Cvqyyni+PlnPwaAWHjpZT19DyKX5N9bXUo8LOaaGHB7VI7OGS\n1C+bdJWaT+q4fUg9m6/TdfuLljy6h39UFLC37AD5ZV9S09jy5nKIJQRHaFyLpB4b7Gu9xwTHYDP5\n06C19Y0cPlbBwaPlfHm0nLLKBv+2HhFBDPtWHAP79mDw9Xb0Rg/hIZLUhRDiWiTJ32Rur5uD5f9g\nb+kBDpw+SL3Hl1AjrOGM7TWSlJhvERcSS2xwDKHWkHaNrdHj5Z/Hqzh4rIIvj5Rz7GSN/5uCIUEa\nw77Vk4F9YxjULwZHjxB5B10IIboISf4mcDbWc+D0IfaVHeDL01/R2NSNHxPcg9EJIxgSN5j+UUkX\nvQd/++3j+etft1/1uHTD4Hhprb8r/+tvKv3vv2uqwrd6RzOwbw8G9Y2hb0IE2iVGHBRCCHHtkuR/\nldS4a8kv+5K9ZQf4quKf/sFjHKF2hsUNZoh9ML3DEwP6AMzV/EjM6aqGpmRfzqFjFdQ4zz5PkBgX\nxsCkGAb168H1vaMJtsk/ByGE6A7kt/0VKG+oYF/Zl+wt209B5VHOjBreOyKRoXGDGRo3mPgwB6tX\nr8ITX0ufjCwAfvvbtSiKwt69e6itrcHj8fDQQ48wbtytVxyTs6GRQ8cqOXisnINHyimpOPtef3S4\njTGD4xnUN4Yb+/YgOjzois8nhBDi2tOtkv+Gf25mT+n+1gteglf34tbduL2N/ta9gkL/qCSGxg1m\nSNxgYkNiWuzzne/cwauvvkJGU/LfsuVvrFixipkz7yM0NJSqqkp+8IMH25T8Gz06/yqu8nflHzlR\nzZn3N4JsGkMH9OTGpq78hNhQ+fSsEEKI7pX828qje3B7G3Hr7hbv2FtVC8lR/Zg9cBZRQRd/1eJb\n37qByspKTp8+RUVFBZGRkcTG9uTVV19m3769qKrCqVNlVFSU06NHzEWPA77vABSV1fmT/VffVOBu\n9MWkKgrJiVEMTOrBoH4x9EuIxKLJfXshhBAtdavkf9eAqdw1YGqr5XRD50hVIXvL9rOv7ADVTV/N\ns6oWUmNvZEjcYFJ73nhZX8WbODGNjz76G6dPnyYt7Q5ycj6kqqqKN954G1VV+d737sTluvB39cur\nGzh4tMLXlX+0guq6s+USYkN9T+T3jeGGPtGEBHWr/6RCCCHawPRMsX37dl544QUMwyAzM5OHH364\nxfbi4mIWLVpEeXk50dHRvPTSSzgcDgBOnDjBs88+y4kTJ1BVlbVr19KrVy9T4vTqXv5RefYd/DMJ\nP1gLZrhjKEPjUhkYewNBzb5xfzluu+12li9/nqqqKl57bS25uX+lR48YVFXliy92c/LkCX9ZwzDY\n83UZB4/4Ev6J02e/mx8VZmP0IAcD+8YwsG8MPSLkvr0QQojLY2ry13WdpUuXsn79eux2O1lZWaSl\npZGcnOwvs2zZMjIyMpg+fTp5eXm88sorLF++HPCN+Pfoo48yevRo6uvrUa/yq2dur5tD5f9gX9mX\n5J86SH3ToDfh1jDGJNzMUPtgru8xAKt65dXUr19/nE4ncXEOYmJiueOOSSxc+AQPPHAPN9yQQkKv\nPuR8VsjxyiJcjV5Wvet7NiHIqnFTciwDk3owsF8MiT3D5L69EEKIK2Jq8s/PzycpKYnExEQApkyZ\nQm5ubovkX1BQwKJFiwAYOXIkjz76qH+9ruuMHj0agJCQq/MBnHpPPQdOHWZv2QEOnj7s/5RudFAU\nI+O/zdC4wfSP6nvVP5UL8F//9T/++aioaH6y6BX+nFfI4W8qCR92C1u/rEVRIH3uq9zYN4ZBfXuQ\nnBgl9+2FEEJcVaYm/5KSEhISEvzLDoeD/ftbPm2fkpJCTk4O999/Pzk5OTidTqqqqjhy5AgRERE8\n9thjFBUVMXr0aJ566qk2tXpr3LXkn2p6B7/87Dv49pCeDLWnMjRuMH0irmvXFnW9y8NrG/ZT1+DB\nERPKoL49GNg3hpQ+0YQGm/sZXyGEEN2bqck/kDGDFixYwNKlS9m4cSPDhw/H4XCgaRper5fPP/+c\n9957j4SEBB5//HE2bNjgH/L3Ys4McHDKWc6u43vZdXwvh0790x9L3+jruPm6YYy8bijXRSZ0WBf6\nO1u+pq7Bw73pKdxzxw0dEsOVaMtAEuLySB2bT+q4fUg9dz6mJv/4+HiKi4v9yyUlJdjt9hZl7HY7\nq1atAsDpdJKTk0N4eDjx8fHceOON/lsGaWlp5OfnXzL5F9eU8NHhPPaWHeBYzTf+9f2jkhjS9NGd\nniGxvpVuOHWq9iJHMpfL7WXDR18TEmRhzI32a+47+fJtf/NJHZtP6rh9SD2br9ON6peamkphYSFF\nRUXExcWRnZ3NihUrWpSpqKggOjoaRVFYs2aNP7mnpqZSXV1NRUUFPXr0YOfOnaSmpl7yfI9/8P8A\n39j2KT2+xZC4wdwUN5DooChTrq+ttu4tosbZyLQxfQkNllfzhBBCtC9TM4+maSxevJi5c+diGAZZ\nWVkkJyezcuVKUlNTmThxIrt27WLFihUoisKIESNYsmQJAKqqsnDhQh544AEABg0axN13333J892e\nfAvxtgQG97yRcGuYmZfWZu5GL3/OKyTIpnH7iN4dHY4QQohuSDECuTF/Dens3Uu5nx/n7b/+g8mj\nksiakNz6Dp2QdOOZT+rYfFLH7UPq2Xxt6faXd8jaUaNH54Odx7BZVe64WVr9QgghOoYk/3b0yf4T\nVNS4mDA0kcjQtn0pUAghhLhSkvzbicfra/VbLSqTRvbp6HCEEEJ0Y5L828mnX57kVFUD44f0Ijpc\nvscvhBCi40jybwdeXSf702NYNIXvSqtfCCFEB5Pk3w52HSyltKKecakJxEQGd3Q4QgghujlJ/ibT\ndYPNnx5FUxUmj0rq6HCEEEIISf5m2/1VKSdOOxk9OJ6e0VdnZEIhhBDiSgSU/LOzs/F4PGbH0uXo\nhsHmHUdRFJgyWlr9QgghOoeAkv/mzZu57bbbePXVVykpKbmsE2zfvp1JkyaRnp7O2rVrz9teXFzM\nnDlzuPPOO5k9e/Z5x6+trWX8+PE8//zzl3XezmDv16c4XlbHqIEOHD1COzocIYQQAggw+a9evZrf\n//73eDweMjMzmT9/Pjt37mx1P13XWbp0KevWrWPz5s1kZ2dTUFDQosyyZcvIyMjgT3/6Ez/60Y94\n5ZVXWmx/9dVXufnmmy/jkjoHwzD40ydHUICpY/p2dDhCCCGEX8D3/BMTE3nyySdZuXIl+fn5/PCH\nP2TatGns3r37ovvk5+eTlJREYmIiVquVKVOmkJub26JMQUEBo0aNAmDkyJEtth84cIDy8nLGjRt3\nudfV4fILTlNYUsuIG+0kxHbOQYaEEEJ0TwElf7fbzXvvvcfMmTN54YUXePzxx8nLy+O5555jwYIF\nF92vpKSEhIQE/7LD4aC0tLRFmZSUFHJycgDIycnB6XRSVVWFYRgsW7aMBQsWcK2NPWQYBpt2HAVg\n6ui+HRqLEEIIca6AhvS97bbbGDlyJD/72c8YNmyYf/3w4cMZPXr0RfcLJGkvWLCApUuXsnHjRoYP\nH47D4UDTNH73u98xYcIEHA5HwMeCto1udLXt+aqUfxVXMzo1gWGDElrf4RrUGeq5q5M6Np/UcfuQ\neu58Akr+GzZswG63X3DbL3/5y4vuFx8fT3FxsX+5pKTkvOPY7XZWrVoFgNPpJCcnh/DwcPbs2cMX\nX3zB7373O+rq6vB4PISFhfHEE09cMtaOHjrSMAze+uAgAHf8n+s6PB4zyBCd5pM6Np/UcfuQejaf\naUP6vvfee1RWVvqXKyoq+M1vftPqfqmpqRQWFlJUVITb7SY7O5u0tLQWZSoqKvyt+jVr1pCZmQnA\nyy+/zJYtW8jNzWXhwoXMmDGj1cTfGXxVWMnXx6sYkhxLUrz8tSuEEKLzCfg9/+joaP9yjx492Lx5\nc6v7aZrG4sWLmTt3LlOnTmXKlCkkJyezcuVKPvroIwB27drFpEmTmDRpEuXl5TzyyCNtvJTO4cy9\n/mlj+3VsIEIIIcRFBNTtf6H77V6vN6ATjB8/nvHjx7dYN3/+fP98eno66enplzxGRkYGGRkZAZ2v\nI319vJJDxyoY1C+G/r0iOzocIYQQ4oICavn37duXN954A8Mw0HWd3/72t/TpI6PTnWvTJ0cBuHNs\n3w6NQwghhLiUgJL/M888w0cffcRNN93E0KFD2bZtG0uWLDE7tmvKv4qrOXCknJQ+0XzruujWdxBC\nCCE6SEDd/g6HgzfffBOn0wlAaKh8qvZcm+VevxBCiGtEQMkfoKamhiNHjuByufzrRowYYUpQ15pj\nJ2vY+89TDLguipQ+0uoXQgjRuQWU/D/44AOWLVtGdXU1drudwsJCUlJS2Lhxo9nxXRPOtPrvHNMX\nRVE6NhghhBCiFQHd8//1r3/Nhg0bSEpK4i9/+Qu/+c1vSE1NNTu2a8Lxslo+/0cZ/RIiGdQvpqPD\nEUIIIVoVUPK3WCzExsb6X+8bO3Ys+/fvNzWwa8XZe/3S6hdCCHFtCKjb32azYRgGSUlJvPXWWyQm\nJvof/uvOTpyu47NDpfRxhDMkObajwxFCCCECElDL/8c//jG1tbU89dRT5Obm8vrrr/Pcc88FdILt\n27czadIk0tPTWbt27Xnbi4uLmTNnDnfeeSezZ8+mpKQEgMOHDzNr1iymTZvG9OnT+eCDDy7jstrH\n5h3HMIBpcq9fCCHENaTVlr/X66WwsJDRo0cTERHB+vXrAz64russXbqU9evXY7fbycrKIi0tjeTk\nZH+ZZcuWkZGRwfTp08nLy+OVV15h+fLlBAcHs3z5cvr06UNpaSl33XUX48ePJzw8vE0XerWVVjjJ\nO1hCYlwYw66P6+hwhBBCiIC12vLXNI0//OEPbTp4fn4+SUlJJCYmYrVamTJlCrm5uS3KFBQUMGrU\nKABGjhzp3963b1//VwTtdjuxsbGUl5e3KQ4zZH96DN0wmDamL6q0+oUQQlxDAur2HzlyJH/+858v\n++AlJSUkJJwdz97hcFBaWtqiTEpKCjk5OQDk5OTgdDqpqqpqUSY/Px+Px9NpPil8qqqeHQdOEh8T\nyvAbLjzUsRBCCNFZBfTA38aNG3njjTcIDg4mJCQEwzBQFIVPP/30kvtdaECgcy1YsIClS5eyceNG\nhg8fjsPhQNM0//bS0lIWLFjA8uXLAwm1TeMaX67/3fYvvLrBPekpOBzdcwCf9qjn7k7q2HxSx+1D\n6rnzCSj5v/vuu206eHx8PMXFxf7lkpIS7PaWLWW73c6qVasAcDqd5OTk+O/r19bW8sgjj/DEE09w\n0003BXTOsrKaNsUaqIoaF3/ddQx7dAgDe0eafr7OKC4uolted3uSOjaf1HH7kHo2X1v+uAoo+Scm\nJl72gQFSU1MpLCykqKiIuLg4srOzWbFiRYsyFRUVREdHoygKa9asITMzE4DGxkZ+9KMfMWPGDO64\n4442nd8MH+48hsdrMGV0Epoa0F0TIYQQolMJKPmPGjXqgq+ytdbtr2kaixcvZu7cuRiGQVZWFsnJ\nyaxcuZLU1FQmTpzIrl27WLFiBYqiMGLECP9ogR9++CGff/451dXVbNiwAUVRePHFF0lJSWnDZV4d\nVbUutu1rsP23AAAVkklEQVQrJjYymNGD4zssDiGEEOJKKEYAN+aLior88y6Xi02bNmGxWPjRj35k\nanBtYWb30h+2fM1fdn3D/ek3MHFY23pDugLpxjOf1LH5pI7bh9Sz+drS7R9Qv3ViYqL/p3///vz4\nxz9m27Ztl32ya1m1081He4roERHEuNSE1ncQQgghOqk23bT+5ptvOH369NWOpVP762ff4G7U+e7I\nPlgtcq9fCCHEteuy7/nruo7H4+GZZ54xNbDOpLa+kdzPjxMZZmP8kF4dHY4QQghxRS77VT+LxULP\nnj1bvIvf1f1t9zc0uL3cObYfNmv3uW4hhBBdU0D913V1dfTo0YPExEQcDgcul4uvv/7a7Ng6BWeD\nh7/uPk54iLVbP+QnhBCi6wgo+f/sZz/DarX6ly0WCwsXLjQtqM4k94vj1Ls8pN/cmyCbtPqFEEJc\n+wJK/l6vt0Xyt9lseL1e04LqLOpdHnJ2FRIWbOG2b1/X0eEIIYQQV0VAyd9isfDNN9/4lwsLC7vF\nPf+te4qoa/Bw+/DehAQF9HiEEEII0ekFlNHmzZvHPffcw6233grAtm3beP755wM6wfbt23nhhRcw\nDIPMzEwefvjhFtuLi4tZtGgR5eXlREdH89JLL+FwOADfgEK//vWvAfjhD3/IjBkzAr6wK+Vq9PKX\nXYWEBGl8Z7i0+oUQQnQdASX/iRMn8t///d988sknADz88MMkJSW1up+u6yxdupT169djt9vJysoi\nLS2N5ORkf5lly5aRkZHB9OnTycvL45VXXmH58uVUVVXx+uuvs3HjRgzD4K677iItLY2IiPYZHWrb\n3mKqnY1MHdOX0GBr6zsIIYQQ14iAuv3Ly8vp1asX9913H/fddx+9evWivLy81f3y8/NJSkoiMTER\nq9XKlClTyM3NbVGmoKCAUaNGATBy5Ej/9r///e+MHTuWiIgIIiMjGTt2LB9//PHlXl+bNHq8fJh3\njCCbxh0jerfLOYUQQoj2ElDy/8EPftDiAb/GxkYeeeSRVvcrKSkhIeHsp3AdDgelpaUtyqSkpJCT\nkwNATk4OTqeTqqqqC+5bUlISSLhXbPu+E1TVurltWCLhIdLqF0II0bUE1O3vdrsJCQnxL4eGhuJy\nuVrdL4Axg1iwYAFLly5l48aNDB8+HIfDgaZpF9z3QiMLnqstAxw01+jR+ctn32Czatz73YFERwRd\n0fG6qiutZ9E6qWPzSR23D6nnzifgR9jLy8uJiYkB4PTp0+i63uo+8fHxFBcX+5dLSkqw2+0tytjt\ndlatWgWA0+kkJyeH8PBw4uPjycvL85c7efKk//bApVzp6FHb9hZxqrKeO0b0prHBTVmD+4qO1xXJ\nKF3mkzo2n9Rx+5B6Np9po/rdf//93HPPPfzHf/wHr7/+Ovfeey8PPPBAq/ulpqZSWFhIUVERbreb\n7Oxs0tLSWpSpqKjwt/LXrFlDZmYmAOPGjWPHjh3U1NRQVVXFjh07GDdu3OVe32XxeHWyPz2GRVOZ\nNLKPqecSQgghOkpALf+srCz69OnD1q1bAfjlL3/J8OHDW91P0zQWL17M3LlzMQyDrKwskpOTWbly\nJampqUycOJFdu3axYsUKFEVhxIgRLFmyBICoqCgeffRRMjMzURSFefPmERkZ2fYrDcDOL0s4VdXA\nbd9OJDpcuvuFEEJ0TYoRwI35mpoa1q5dy+HDh2loaPDfe3/zzTdND/BytbV7SdcNnvnPnZyqamDZ\nI6OJiQy+ypF1HdKNZz6pY/NJHbcPqWfzmdbtv2jRIjRN4+jRo8ycORNN07jpppsu+2Sd2a5DJZRU\n1DPupgRJ/EIIIbq0gJL/sWPHePzxxwkODmbq1KmsWbOG3bt3mx1bu9ENg007jqIqCpNHtf7xIiGE\nEOJaFlDyt9lsAFitViorK7FarQF95Oda8flXZZw47WT0YAdx0SGt7yCEEEJcwwJ64K9v375UVlYy\nbdo0Zs6cSUREBIMGDTI7tnahGwabPjmKosDU0X07OhwhhBDCdAEl/5dffhmABx98kNTUVGpqahg/\nfrypgbWXfV+f4nhZLaMGOXDEhHZ0OEIIIYTpLnuc2kBe8btWGIbBn3YcRUFa/UIIIbqPgO75d1X7\n/1XOsZM1/J8UO716hnV0OEIIIUS76LbJ3zAMNn1yBIBpY/p2bDBCCCFEO+q2yf/gsQoKiqsZ9q2e\n9LaHd3Q4QgghRLsxPflv376dSZMmkZ6eztq1a8/bfuLECWbPnk1GRgbTp09n27ZtAHg8Hn72s58x\nbdo0pkyZcsF9r8SmT44CMG1s36t6XCGEEKKzu+wH/i6HrussXbqU9evXY7fbycrKIi0tjeTkZH+Z\n1atXM3nyZGbNmkVBQQEPPfQQW7Zs4c9//jONjY1s2rSJhoYGJk+ezNSpU+nVq9cVx/VVYQX/+KaS\nm5Jj6Rtv7ngBQgghRGdjass/Pz+fpKQkEhMTsVqtTJkyhdzc3BZlFEWhtrYWgOrqahwOh3+90+nE\n6/VSX1+PzWYjPPzqdM//6UyrX+71CyGE6IZMbfmXlJSQkJDgX3Y4HOzfv79FmXnz5jF37lzeeust\nGhoaeOONNwBIT08nNzeXcePG0dDQwKJFi67KqH7/LKri0LEKBvXtQXJi1BUfTwghhLjWmJr8Axgw\nkOzsbDIzM5kzZw579+7lpz/9KdnZ2ezbtw9N0/jkk0+orKzk3nvvZfTo0Vx33XWXPF5roxu9/t4B\nAO6fMqhNIyEJH6k780kdm0/quH1IPXc+pib/+Ph4iouL/cslJSXY7fYWZd555x3WrVsHwNChQ3G7\n3ZSXl5Odnc0tt9yCqqrExMTw7W9/mwMHDrSa/C81dOSRE9V8friUG3pHY4+wyTCTbSRDdJpP6th8\nUsftQ+rZfKYN6dtWqampFBYWUlRUhNvtJjs7m7S0tBZlevXqxY4dOwAoKCjA5XIRExNDQkICO3fu\nBMDpdLJv3z769+9/RfHIE/5CCCGEyS1/TdNYvHgxc+fOxTAMsrKySE5OZuXKlaSmpjJx4kQWLlzI\ns88+y/r161FVlWXLlgFw33338fTTTzN16lQAsrKyuP7669scS2FJDXv/eYoBiVHcmNTjqlyfEEII\ncS1SjEBuzF9DLta99B8b97P7qzJ+cvcQUvvHtnNUXYt045lP6th8UsftQ+rZfJ2u27+zKCqr5fOv\nyugbH8HgfjEdHY4QQgjRobpF8t/86TEMfPf6FUXp6HCEEEKIDtXlk//Jcie7DpXQ2x7O0AE9Ozoc\nIYQQosN1+eSfveMohuH7mp+0+oUQQogunvxLK+v59MsSEnuG8e0b4jo6HCGEEKJT6NLJ/4NPj6Ib\nBlPGJKFKq18IIYQAunDyP13VwCf7T+KICeXmFEdHhyOEEEJ0Gl02+X+QdwyvbjB1dBKqKq1+IYQQ\n4gzTk//27duZNGkS6enprF279rztJ06cYPbs2WRkZDB9+nS2bdvm33b48GFmzZrF1KlTufPOO3G7\n3QGds6LGxcf7iomLDmbUIGn1CyGEEM2Z+nlfXddZunQp69evx263k5WVRVpaGsnJyf4yq1evZvLk\nycyaNYuCggIeeughtmzZgtfrZcGCBbz88stcf/31VFVVYbVaAzrvh3nH8HgNpozui6Z22c4NIYQQ\nok1MzYz5+fkkJSWRmJiI1WplypQp5ObmtiijKAq1tbUAVFdX43D4Wup///vfSUlJ8X/PPyoqKqBX\n9arq3GzbW0xsZBBjBsdf5SsSQgghrn2mtvxLSkpISEjwLzscDvbv39+izLx585g7dy5vvfUWDQ0N\nvPHGGwAcPXoUgO9///tUVFQwefJk/u3f/q3Vc/5lVyGNHp3Jo5KwaNLqF0IIIc5lavIPZMyg7Oxs\nMjMzmTNnDnv37uWnP/0p2dnZeL1evvjiC959912CgoKYM2cOgwcPZtSoURc9VlWti617ioiJDGbG\nbddjs2pX83JEM20ZSEJcHqlj80kdtw+p587H1OQfHx9PcXGxf7mkpAS73d6izDvvvMO6desAGDp0\nKC6Xi/LycuLj4xkxYgRRUVEAjB8/noMHD14y+b+/vYAGt5eMW/pTVek04YoEyChd7UHq2HxSx+1D\n6tl8nW5Uv9TUVAoLCykqKsLtdpOdnU1aWlqLMr169WLHjh0AFBQU4Ha7iYmJYdy4cXz11Ve4XC48\nHg+fffZZiwcFLyT7kyNEhtkYP7SXadckhBBCXOtMbflrmsbixYuZO3cuhmGQlZVFcnIyK1euJDU1\nlYkTJ7Jw4UKeffZZ1q9fj6qqLFu2DIDIyEgefPBBMjMzURSFCRMmcOutt17yfM4GD3dPHECQdPcL\nIYQQF6UYgdyYv0Zs33Oc63tFyOt9JpNuPPNJHZtP6rh9SD2br9N1+7e38cOuk8QvhBBCtEIypRBC\nCNHNSPIXQgghuhlJ/kIIIUQ3I8lfCCGE6GYk+QshhBDdjCR/IYQQopuR5C+EEEJ0M5L8hRBCiG7G\n9OS/fft2Jk2aRHp6OmvXrj1v+4kTJ5g9ezYZGRlMnz6dbdu2tdheXFzMsGHD/EP9CiGEEOLKmPpt\nf13XWbp0KevXr8dut5OVlUVaWlqLAXpWr17N5MmTmTVrFgUFBTz00ENs2bLFv/1Xv/pVq9/0F0II\nIUTgTG355+fnk5SURGJiIlarlSlTppCbm9uijKIo1NbWAlBdXY3D4fBv+9vf/kbv3r0ZMGCAmWEK\nIYQQ3Yqpyb+kpISEhAT/ssPhoLS0tEWZefPm8f7773PrrbfyyCOPsHjxYgDq6+v5zW9+w7x588wM\nUQghhOh2TO32D2TAwOzsbDIzM5kzZw579+7lpz/9KdnZ2axcuZI5c+YQEhIS8LGgbaMbicsn9Ww+\nqWPzSR23D6nnzsfU5B8fH09xcbF/uaSkBLvd3qLMO++8w7p16wAYOnQobreb8vJy8vPzycnJ4aWX\nXqK6uhpVVQkKCuK+++4zM2QhhBCiyzM1+aemplJYWEhRURFxcXFkZ2ezYsWKFmV69erFjh07yMjI\noKCgAJfLRUxMDG+//ba/zGuvvUZYWJgkfiGEEOIqMDX5a5rG4sWLmTt3LoZhkJWVRXJyMitXriQ1\nNZWJEyeycOFCnn32WdavX4+qqixbtszMkIQQQohuTzECvZkuhBBCiC5BvvAnhBBCdDOS/IUQQohu\nRpK/EEII0c10meTf2hgC4sqcPHmS2bNnM3nyZKZNm8abb77Z0SF1Wbquk5GRwSOPPNLRoXRZNTU1\nzJ8/n+9+97tMmTKFffv2dXRIXc769euZOnUq06ZN48knn8Ttdnd0SF3CokWLGDNmDNOmTfOvq6qq\nYu7cuaSnp/P973+fmpqaVo/TJZL/mTEE1q1bx+bNm8nOzqagoKCjw+pSNE3j6aef5oMPPuB//ud/\nePvtt6WOTfLmm2+2GP9CXH2//OUvufXWW/nwww95//33pb6vspKSEt566y02bNjApk2b8Hq9fPDB\nBx0dVpdw1113+b+Nc8batWsZPXo0f/nLXxg5ciRr1qxp9ThdIvkHMoaAuDJxcXHceOONAISFhZGc\nnHzep5rFlTt58iTbtm3je9/7XkeH0mXV1taye/duMjMzAbBYLISHh3dwVF2PruvU19fj8XhoaGg4\n7wNvom2GDx9OZGRki3W5ublkZGQAkJGRwd/+9rdWj9Mlkn8gYwiIq+f48eMcPnyYm266qaND6XJe\neOEFFixYgKIoHR1Kl3X8+HF69OjB008/TUZGBosXL6ahoaGjw+pSHA4HDz74IBMmTGD8+PFEREQw\nZsyYjg6ryyovL6dnz56Ar6FWUVHR6j5dIvnLpwraT11dHfPnz2fRokWEhYV1dDhdytatW+nZsyc3\n3nij/Js2kcfj4eDBg9x7771s3LiR4OBgeU7oKquuriY3N5ePPvqIjz/+GKfTyaZNmzo6LNFMl0j+\ngYwhIK6cx+Nh/vz5TJ8+ne985zsdHU6X88UXX7BlyxbS0tJ48sknycvLY8GCBR0dVpcTHx9PfHw8\nqampAKSnp3Pw4MEOjqpr2bFjB7179yY6OhpN07j99tvZs2dPR4fVZcXGxnLq1CkAysrKiImJaXWf\nLpH8m48h4Ha7yc7OJi0traPD6nIWLVrEgAEDeOCBBzo6lC7piSeeYOvWreTm5rJixQpGjhzJ8uXL\nOzqsLqdnz54kJCRw5MgRAHbu3CkP/F1lvXr1Yt++fbhcLgzDkDq+ys7tGbztttvYsGEDABs3bgwo\n/5n6bf/2crExBMTV8/nnn7Np0yauv/56ZsyYgaIo/OQnP2H8+PEdHZoQl+3ZZ5/lqaeewuPx0Lt3\nb1588cWODqlLuemmm0hPT2fGjBlYLBYGDhzI3Xff3dFhdQlnegUrKyuZMGECjz32GA8//DA//vGP\neffdd+nVqxevvvpqq8eRb/sLIYQQ3UyX6PYXQgghROAk+QshhBDdjCR/IYQQopuR5C+EEEJ0M5L8\nhRBCiG5Gkr8QQgjRzUjyF0K0m127dvkH1BFCdBxJ/kKIdiWDFgnR8brEF/6EEFcuPz+fl19+mbq6\nOgDmz5/PgAEDyMzMJCMjg08++QSAJUuWMHz4cADee+891q1bh6qq9OnTh5///Of+74qvWbOGzZs3\no6oqoaGh/P73vwd8Y0QsWbKEvXv3oqoqK1asoH///h1wxUJ0Y4YQoturrq42ZsyYYZSVlRmGYRil\npaXG+PHjjUOHDhk33HCD8f777xuGYRi7du0yxo8fb7jdbuOrr74yxo0bZ5w6dcowDMP493//d+Px\nxx83DMMwNmzYYMycOdOoq6szDMMwKisrDcMwjLy8PGPQoEHGoUOHDMMwjNWrVxtPPfVUu16rEMIw\npOUvhOCLL77g+PHjPPTQQ/5BQzRNw+PxYLPZuPPOOwEYMWIEwcHBHDlyhF27djFhwgRiY2MBmDVr\nFtOnTwd8wxPfc889hIaGAhAVFeU/V79+/UhJSQFgyJAhbN26tb0uUwjRRJK/EAKAlJQU3nrrrRbr\nioqKzitnGAaKovin564/M38xQUFB/vkzf2AIIdqXPPAnhGDYsGEcPXqUvLw8/7r9+/djGAZut5tN\nmzYBsHv3btxuN/369WP06NFs27aN06dPA/DHP/6RMWPGAL4hRn//+9/7nx+orKxs5ysSQlyKtPyF\nEERGRrJ69WqWLVvGiy++iNvtpk+fPjzzzDNER0dz6NAh/vM//xOAFStWYLFYGDBgAE888QRz5sxB\nVVV69+7NL37xCwBmzJhBaWkpM2fORNM0wsPDefvttzvyEoUQzciQvkKIiyoqKiIzM5OdO3d2dChC\niKtIuv2FEJck7+UL0fVIy18IIYToZqTlL4QQQnQzkvyFEEKIbkaSvxBCCNHNSPIXQgghuhlJ/kII\nIUQ38/8BPlWMilIodCgAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<matplotlib.figure.Figure at 0x563d01086bd0>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "CPU times: user 23min 48s, sys: 1min 7s, total: 24min 55s\n",
            "Wall time: 3min 54s\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "H-GfLCrGUwsD",
        "colab_type": "text"
      },
      "source": [
        "## Exercise:  Try out different cells!\n",
        "\n",
        "Replace the BasicLSTM cell with something else."
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {
        "id": "HVLm6vflU5v8",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "# Your code here..."
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "hIAnbWvA-uDq",
        "colab_type": "text"
      },
      "source": [
        "## Known good settings\n",
        "\n",
        "We got **98.8%** with this model and hyperparams:"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {
        "id": "btJIqQJ5-05c",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "# Specify the model hyperparameters:\n",
        "model_params = {\n",
        "    'input_size' : 784,\n",
        "    'output_size' : 10,\n",
        "    'batch_size'  : 100,\n",
        "    'num_steps' : 28,          \n",
        "    'pixels_per_step' : 28,  # NOTE: num_steps * pixels_per_step must = input_size\n",
        "    'cell_fn' : tf.contrib.rnn.BasicLSTMCell,\n",
        "    'memory_units' : 128,\n",
        "    'num_layers' : 1,\n",
        "    'l2_lambda' : 1e-3,\n",
        "    'dropout_keep_prob': 1.\n",
        "}\n",
        "\n",
        "\n",
        "# Specify the training hyperparameters:\n",
        "training_params = {\n",
        "    'num_epochs' : 100,   # Max epochs/iterations to train for.\n",
        "    'batch_size' : 100,  # Number of examples per batch, 100 default.\n",
        "    #'keep_prob' : 1.0,  # (1. - dropout) probability, none by default.\n",
        "    'train_only_on_fraction' : 1.,  # Fraction of training data to use, 1. for everything.\n",
        "    'optimizer_fn' : None,  # Optimizer, None for Adam.\n",
        "    'report_every' : 1, # Report training results every nr of epochs.\n",
        "    'eval_every' : 1,   # Evaluate on validation data every nr of epochs.\n",
        "   'stop_early' : True,  # Use early stopping or not.\n",
        "}\n",
        "\n",
        "# Build, train, evaluate and plot the results!\n",
        "trained_model, training_results = build_train_eval_and_plot(\n",
        "    model_params, \n",
        "    training_params, \n",
        "    verbose=True  # Modify as desired.\n",
        ")\n"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "HEHT4Ne4b07g",
        "colab_type": "text"
      },
      "source": [
        "# Resources\n",
        "\n",
        "* https://r2rt.com/written-memories-understanding-deriving-and-extending-the-lstm.html\n",
        "* HipsterNet Code: https://github.com/wiseodd/hipsternet/blob/master/hipsternet/neuralnet.py\n",
        "* http://peterroelants.github.io/posts/rnn_implementation_part01/\n",
        "* http://karpathy.github.io/2015/05/21/rnn-effectiveness/\n"
      ],
      "cell_type": "markdown"
    }
  ]
}