02b_advanced_networks_solution.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Music machine learning - Advanced neural networks\n",
    "\n",
    "### Author: Philippe Esling (esling@ircam.fr)\n",
    "\n",
    "In this course we will cover\n",
    "1. An introduction to [convolutions](#convolution) and how they can be used\n",
    "2. Defining a [Convolutional Neural Network](#cnn) in Pytorch for image classification \n",
    "3. Coding our own [convolutional layer](#layer)\n",
    "4. An explanation on [recurrent networks](#rnn) in practice"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"convolution\"></a>\n",
    "## Convolutions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In purely mathematical terms, convolution is a function derived from two given functions by integration which expresses how the shape of one is modified by the other. In simpler (discrete) terms, the convolution product of a matrix by a smaller one can be seen as _filtering_ the large matrix. Hence we slide the small matrix over the large one and compute local products at each position. Therefore the convolution operator $\\star$ computes at each position $n$\n",
    "\n",
    "$$\n",
    "(f \\star g)[n]=\\sum _{m=-M}^{M}f[n-m]g[m].\n",
    "$$\n",
    "\n",
    "An example of this operation is shown here\n",
    "\n",
    "<img src=\"images/02_convolution.png\" align=\"center\"/>\n",
    "\n",
    "This operation can be used to _filter_ the image (as in the _gaussian blur_ operator), or _detect_ features (such as edges). \n",
    "\n",
    "Given an 32x32 image with RGB channels, we can represent it as a tensor of shape `(32, 32, 3)` which is (height, width, channels). When we perform convolution, we need a filter that has the same channel depth as the image. For example, we can use a 5x5 filter which is of shape `(5, 5, 3)` and slide it across the image left to right, top to bottom with a stride of 1 to perform convolution. We are going to perform this in numpy, depending on a certain amount of parameters, which define the behavior of our convolution\n",
    "\n",
    "* `height` and `width`: spatial extend of the filters\n",
    "* `S`: stride size (number of steps to jump to the next position)\n",
    "* `P`: amount of padding (adding zeros in the original matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "# Padding\n",
    "pad = 2\n",
    "stride = 1\n",
    "height, width = 5, 5\n",
    "# A random fake image\n",
    "x = np.random.randn(3, 32, 32)\n",
    "x_h, x_w = x.shape[1:]\n",
    "# Our convolution kernel\n",
    "weight = np.random.randn(3, 5, 5)\n",
    "# Padding the original image\n",
    "x_pad = np.pad(x, pad_width=((0, 0,), (pad, pad), (pad, pad)), mode='constant', constant_values=0)\n",
    "# We can expect the output size to be\n",
    "h_out = int(1 + (x_h + 2 * pad - height) / stride)\n",
    "w_out = int(1 + (x_w + 2 * pad - width) / stride)\n",
    "# So we will store our result in\n",
    "y = np.zeros((1, h_out, w_out))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Computing the convolution itself can be performed by using the following loop (which amounts to _slide_ our kernel across the large matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sliding the kernel across the matrix\n",
    "for h in range(h_out):\n",
    "    for w in range(w_out):\n",
    "        i, j = h * stride, w * stride\n",
    "        conv_sum = np.sum(x_pad[:, i:i+height, j:j+width] * weight)\n",
    "        y[0, h, w] = conv_sum"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can witness the effect of this operation with the following plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convolutional Neural Networks (CNN)\n",
    "\n",
    "Convolutional NNs (CNNs) rely on convolution in place of general matrix multiplication. They are specialized for processing data with a known grid-like topology and are among the best performing systems in classification/recognition tasks. Each layer in a CNN consists in a set of $N$ _filters_ called _kernels_, that are convolved across the input. If we denote as $\\{k^l_n\\}_{n\\in[1;N]}$ the set of kernels for layer $l$, these all share a unique _kernel size_. By convolving each one of its $N$ kernels across a d-dimensional input $x$, a convolutional layer produces $N$ d-dimensional outputs called _feature maps_, denoted as $\\{a^l_n\\}_{n\\in[1;N]}$. Hence, the computation of the $n$-th activation map in layer $l$ for input $x$ is defined as:\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "a^l_m = \\sum_{n=1}^{N} k^l_m \\star x_n + b^l_m\n",
    "\\end{equation}\n",
    "$$\n",
    "\n",
    "Thus, as depicted in the following Figure, the feature map corresponding to kernel $n$ consists in the sum of the d-dimensional discrete convolutions (denoted by the $\\star$ operator) between the kernel $n$ and each one of the d-dimensional data $\\{x_m\\}_{m\\in[1;M]}$, plus a bias $b$. A convolutional layer is thus a 3-dimensional tensor $h \\in \\mathcal{T}_{N,I,J}(\\mathbb{R})$ where $N$ is the number of features maps while $I$ and $J$ are respectively the _width_ and _height_ of the maps. \n",
    "\n",
    "<img src=\"images/02_cnns.png\" align=\"center\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This can be seen as replacing our _neurons_ by _feature detectors_ (the convolutional kernels), which will increasingly process the image. In the following, we will first use the high-level interface of `Pytorch` to define a CNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"cnn\"></a>\n",
    "## Defining a CNN in Pytorch\n",
    "\n",
    "Defining a convolutional network in Pytorch is quite easy, as we can rely on the `nn` module, which contains all the required layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the previous course, we have seen that we could define our network in a very simple way, by using the `Sequential` model definition. Here we define a CNN followed by a MLP, as seen in the previous course"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.0921, 0.1126, 0.1053, 0.0997, 0.1114, 0.0908, 0.0943, 0.0938, 0.1048,\n",
       "         0.0951]], grad_fn=<SoftmaxBackward0>)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Define the input dimensions\n",
    "in_size = 1000\n",
    "# Use the nn package to define our model and loss function.\n",
    "model = nn.Sequential(\n",
    "    nn.Conv2d(3, 6, 5),\n",
    "    nn.ReLU(),\n",
    "    nn.MaxPool2d(2, 2),\n",
    "    nn.Conv2d(6, 16, 5),\n",
    "    nn.ReLU(),\n",
    "    nn.MaxPool2d(2, 2),\n",
    "    nn.Flatten(),\n",
    "    nn.Linear(16 * 5 * 5, 120),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(120, 84),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(84, 10),\n",
    "    nn.Softmax(dim=1)\n",
    ")\n",
    "# Batch x Channel x H x L\n",
    "inputs = torch.rand(16, 3, 32, 32)\n",
    "#model(inputs)\n",
    "type(model)\n",
    "model(inputs[0].unsqueeze(0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(16.0000)"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Define the input dimensions\n",
    "in_size = 10\n",
    "# Use the nn package to define our model and loss function.\n",
    "model = nn.Sequential(\n",
    "    nn.Linear(10, 100),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(100, 2),\n",
    "    nn.Softmax(dim=1)\n",
    ")\n",
    "# Batch x Channel x H x L\n",
    "inputs = torch.rand(16, in_size)\n",
    "model(inputs)\n",
    "\n",
    "# Functional\n",
    "import torch.nn.functional as F\n",
    "\n",
    "class MonReseauToutMignon(nn.Module):\n",
    "\n",
    "    def __init__(self):\n",
    "        super(MonReseauToutMignon, self).__init__()\n",
    "        self.lin1 = nn.Linear(10, 100)\n",
    "        self.lin2 = nn.Linear(100, 2)\n",
    "    \n",
    "    def forward(self, inputs):\n",
    "        # Same as sequential above\n",
    "        out_l = lin1(inputs)\n",
    "        out_r = F.relu(out_l)\n",
    "        out_s = F.sigmoid(out_l)\n",
    "        out_add = out_s + out_r\n",
    "        out = lin2(out_add)\n",
    "        out = F.softmax(inputs, dim=1)\n",
    "        out = out.sum()\n",
    "        return out\n",
    "    \n",
    "reseau = MonReseauToutMignon()\n",
    "reseau(inputs)\n",
    "res2 = nn.Sequential(\n",
    "    MonReseauToutMignon(),\n",
    "    nn.ReLU()\n",
    ")\n",
    "res2(inputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another way to implement the networks in Pytorch is to use the `functional` approach. In this version, each layer is seen as a function, that we apply on sucessive inputs. For instance, we can define one layer of fully-connected units and apply it to some inputs as follows.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0.0000e+00, 0.0000e+00, 8.9303e-02, 1.7173e-02, 0.0000e+00, 7.0042e-02,\n",
      "         0.0000e+00, 1.6613e-01, 0.0000e+00, 0.0000e+00],\n",
      "        [0.0000e+00, 0.0000e+00, 0.0000e+00, 4.2463e-01, 0.0000e+00, 0.0000e+00,\n",
      "         0.0000e+00, 3.7440e-01, 0.0000e+00, 0.0000e+00],\n",
      "        [0.0000e+00, 1.5053e-01, 0.0000e+00, 1.0441e-01, 0.0000e+00, 0.0000e+00,\n",
      "         0.0000e+00, 2.6158e-01, 0.0000e+00, 0.0000e+00],\n",
      "        [0.0000e+00, 6.4797e-02, 0.0000e+00, 2.1967e-01, 0.0000e+00, 0.0000e+00,\n",
      "         0.0000e+00, 8.0643e-02, 0.0000e+00, 0.0000e+00],\n",
      "        [0.0000e+00, 0.0000e+00, 0.0000e+00, 2.0422e-01, 0.0000e+00, 0.0000e+00,\n",
      "         0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00],\n",
      "        [0.0000e+00, 0.0000e+00, 1.6462e-05, 2.7618e-01, 0.0000e+00, 0.0000e+00,\n",
      "         0.0000e+00, 4.9077e-01, 0.0000e+00, 0.0000e+00],\n",
      "        [0.0000e+00, 0.0000e+00, 1.6345e-01, 4.0888e-01, 0.0000e+00, 2.0245e-01,\n",
      "         0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00],\n",
      "        [0.0000e+00, 0.0000e+00, 6.7304e-02, 4.2857e-01, 0.0000e+00, 0.0000e+00,\n",
      "         0.0000e+00, 4.3335e-02, 4.8058e-02, 0.0000e+00],\n",
      "        [0.0000e+00, 9.5278e-02, 5.7948e-02, 5.1186e-01, 0.0000e+00, 0.0000e+00,\n",
      "         0.0000e+00, 3.3434e-01, 1.7896e-01, 0.0000e+00],\n",
      "        [0.0000e+00, 0.0000e+00, 0.0000e+00, 2.4809e-01, 0.0000e+00, 0.0000e+00,\n",
      "         0.0000e+00, 2.0784e-02, 1.8756e-02, 0.0000e+00],\n",
      "        [0.0000e+00, 0.0000e+00, 4.9172e-02, 2.2616e-02, 0.0000e+00, 8.9603e-03,\n",
      "         0.0000e+00, 2.1834e-01, 0.0000e+00, 0.0000e+00],\n",
      "        [0.0000e+00, 0.0000e+00, 0.0000e+00, 2.8372e-01, 0.0000e+00, 0.0000e+00,\n",
      "         0.0000e+00, 2.2422e-01, 0.0000e+00, 0.0000e+00],\n",
      "        [0.0000e+00, 6.6453e-02, 1.8000e-01, 2.9409e-01, 0.0000e+00, 1.3128e-01,\n",
      "         0.0000e+00, 7.4493e-02, 0.0000e+00, 0.0000e+00],\n",
      "        [0.0000e+00, 4.5365e-03, 0.0000e+00, 2.8036e-01, 0.0000e+00, 2.4960e-01,\n",
      "         0.0000e+00, 1.5819e-01, 0.0000e+00, 0.0000e+00],\n",
      "        [0.0000e+00, 0.0000e+00, 0.0000e+00, 2.6192e-01, 0.0000e+00, 0.0000e+00,\n",
      "         0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00],\n",
      "        [0.0000e+00, 0.0000e+00, 9.3908e-02, 2.4551e-01, 0.0000e+00, 1.7683e-02,\n",
      "         0.0000e+00, 1.9090e-01, 1.0616e-01, 0.0000e+00],\n",
      "        [0.0000e+00, 0.0000e+00, 6.9130e-02, 2.1738e-01, 0.0000e+00, 2.8617e-02,\n",
      "         0.0000e+00, 1.2053e-01, 0.0000e+00, 0.0000e+00],\n",
      "        [0.0000e+00, 0.0000e+00, 0.0000e+00, 3.3173e-01, 0.0000e+00, 0.0000e+00,\n",
      "         0.0000e+00, 3.4309e-03, 0.0000e+00, 0.0000e+00],\n",
      "        [0.0000e+00, 0.0000e+00, 0.0000e+00, 2.6944e-01, 0.0000e+00, 5.7483e-02,\n",
      "         0.0000e+00, 2.1273e-01, 0.0000e+00, 0.0000e+00],\n",
      "        [0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 9.6322e-02,\n",
      "         0.0000e+00, 2.3808e-01, 0.0000e+00, 0.0000e+00],\n",
      "        [0.0000e+00, 2.6554e-01, 0.0000e+00, 6.8435e-02, 0.0000e+00, 1.1546e-01,\n",
      "         0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00],\n",
      "        [0.0000e+00, 0.0000e+00, 0.0000e+00, 3.5302e-02, 0.0000e+00, 4.0776e-02,\n",
      "         0.0000e+00, 7.1194e-02, 0.0000e+00, 0.0000e+00],\n",
      "        [0.0000e+00, 0.0000e+00, 0.0000e+00, 2.3869e-01, 0.0000e+00, 0.0000e+00,\n",
      "         0.0000e+00, 1.1618e-01, 0.0000e+00, 0.0000e+00],\n",
      "        [0.0000e+00, 0.0000e+00, 0.0000e+00, 1.3691e-01, 0.0000e+00, 1.3287e-01,\n",
      "         0.0000e+00, 2.3332e-01, 0.0000e+00, 0.0000e+00],\n",
      "        [0.0000e+00, 7.2005e-02, 0.0000e+00, 2.1099e-01, 0.0000e+00, 0.0000e+00,\n",
      "         0.0000e+00, 2.1895e-01, 0.0000e+00, 0.0000e+00],\n",
      "        [0.0000e+00, 0.0000e+00, 0.0000e+00, 1.5570e-01, 0.0000e+00, 2.7824e-02,\n",
      "         0.0000e+00, 1.9135e-01, 8.1555e-02, 0.0000e+00],\n",
      "        [0.0000e+00, 4.2206e-02, 0.0000e+00, 2.0340e-01, 0.0000e+00, 0.0000e+00,\n",
      "         0.0000e+00, 1.8679e-01, 0.0000e+00, 0.0000e+00],\n",
      "        [0.0000e+00, 0.0000e+00, 1.7033e-02, 3.1707e-01, 0.0000e+00, 1.8956e-01,\n",
      "         6.2846e-02, 4.1277e-02, 0.0000e+00, 0.0000e+00],\n",
      "        [0.0000e+00, 0.0000e+00, 7.4989e-02, 3.1062e-01, 0.0000e+00, 0.0000e+00,\n",
      "         0.0000e+00, 1.7190e-02, 0.0000e+00, 0.0000e+00],\n",
      "        [0.0000e+00, 0.0000e+00, 0.0000e+00, 1.1783e-01, 0.0000e+00, 1.7702e-01,\n",
      "         0.0000e+00, 1.8177e-01, 7.2158e-02, 0.0000e+00],\n",
      "        [0.0000e+00, 0.0000e+00, 0.0000e+00, 2.7038e-01, 0.0000e+00, 8.5636e-02,\n",
      "         0.0000e+00, 2.6861e-01, 0.0000e+00, 0.0000e+00],\n",
      "        [0.0000e+00, 0.0000e+00, 1.3341e-01, 3.3453e-01, 0.0000e+00, 0.0000e+00,\n",
      "         0.0000e+00, 4.9566e-01, 0.0000e+00, 0.0000e+00]],\n",
      "       grad_fn=<ReluBackward0>)\n",
      "tensor([[3.0000, 3.0000, 3.0893, 3.0172, 3.0000, 3.0700, 3.0000, 3.1661, 3.0000,\n",
      "         3.0000],\n",
      "        [3.0000, 3.0000, 3.0000, 3.4246, 3.0000, 3.0000, 3.0000, 3.3744, 3.0000,\n",
      "         3.0000],\n",
      "        [3.0000, 3.1505, 3.0000, 3.1044, 3.0000, 3.0000, 3.0000, 3.2616, 3.0000,\n",
      "         3.0000],\n",
      "        [3.0000, 3.0648, 3.0000, 3.2197, 3.0000, 3.0000, 3.0000, 3.0806, 3.0000,\n",
      "         3.0000],\n",
      "        [3.0000, 3.0000, 3.0000, 3.2042, 3.0000, 3.0000, 3.0000, 3.0000, 3.0000,\n",
      "         3.0000],\n",
      "        [3.0000, 3.0000, 3.0000, 3.2762, 3.0000, 3.0000, 3.0000, 3.4908, 3.0000,\n",
      "         3.0000],\n",
      "        [3.0000, 3.0000, 3.1634, 3.4089, 3.0000, 3.2024, 3.0000, 3.0000, 3.0000,\n",
      "         3.0000],\n",
      "        [3.0000, 3.0000, 3.0673, 3.4286, 3.0000, 3.0000, 3.0000, 3.0433, 3.0481,\n",
      "         3.0000],\n",
      "        [3.0000, 3.0953, 3.0579, 3.5119, 3.0000, 3.0000, 3.0000, 3.3343, 3.1790,\n",
      "         3.0000],\n",
      "        [3.0000, 3.0000, 3.0000, 3.2481, 3.0000, 3.0000, 3.0000, 3.0208, 3.0188,\n",
      "         3.0000],\n",
      "        [3.0000, 3.0000, 3.0492, 3.0226, 3.0000, 3.0090, 3.0000, 3.2183, 3.0000,\n",
      "         3.0000],\n",
      "        [3.0000, 3.0000, 3.0000, 3.2837, 3.0000, 3.0000, 3.0000, 3.2242, 3.0000,\n",
      "         3.0000],\n",
      "        [3.0000, 3.0665, 3.1800, 3.2941, 3.0000, 3.1313, 3.0000, 3.0745, 3.0000,\n",
      "         3.0000],\n",
      "        [3.0000, 3.0045, 3.0000, 3.2804, 3.0000, 3.2496, 3.0000, 3.1582, 3.0000,\n",
      "         3.0000],\n",
      "        [3.0000, 3.0000, 3.0000, 3.2619, 3.0000, 3.0000, 3.0000, 3.0000, 3.0000,\n",
      "         3.0000],\n",
      "        [3.0000, 3.0000, 3.0939, 3.2455, 3.0000, 3.0177, 3.0000, 3.1909, 3.1062,\n",
      "         3.0000],\n",
      "        [3.0000, 3.0000, 3.0691, 3.2174, 3.0000, 3.0286, 3.0000, 3.1205, 3.0000,\n",
      "         3.0000],\n",
      "        [3.0000, 3.0000, 3.0000, 3.3317, 3.0000, 3.0000, 3.0000, 3.0034, 3.0000,\n",
      "         3.0000],\n",
      "        [3.0000, 3.0000, 3.0000, 3.2694, 3.0000, 3.0575, 3.0000, 3.2127, 3.0000,\n",
      "         3.0000],\n",
      "        [3.0000, 3.0000, 3.0000, 3.0000, 3.0000, 3.0963, 3.0000, 3.2381, 3.0000,\n",
      "         3.0000],\n",
      "        [3.0000, 3.2655, 3.0000, 3.0684, 3.0000, 3.1155, 3.0000, 3.0000, 3.0000,\n",
      "         3.0000],\n",
      "        [3.0000, 3.0000, 3.0000, 3.0353, 3.0000, 3.0408, 3.0000, 3.0712, 3.0000,\n",
      "         3.0000],\n",
      "        [3.0000, 3.0000, 3.0000, 3.2387, 3.0000, 3.0000, 3.0000, 3.1162, 3.0000,\n",
      "         3.0000],\n",
      "        [3.0000, 3.0000, 3.0000, 3.1369, 3.0000, 3.1329, 3.0000, 3.2333, 3.0000,\n",
      "         3.0000],\n",
      "        [3.0000, 3.0720, 3.0000, 3.2110, 3.0000, 3.0000, 3.0000, 3.2189, 3.0000,\n",
      "         3.0000],\n",
      "        [3.0000, 3.0000, 3.0000, 3.1557, 3.0000, 3.0278, 3.0000, 3.1914, 3.0816,\n",
      "         3.0000],\n",
      "        [3.0000, 3.0422, 3.0000, 3.2034, 3.0000, 3.0000, 3.0000, 3.1868, 3.0000,\n",
      "         3.0000],\n",
      "        [3.0000, 3.0000, 3.0170, 3.3171, 3.0000, 3.1896, 3.0628, 3.0413, 3.0000,\n",
      "         3.0000],\n",
      "        [3.0000, 3.0000, 3.0750, 3.3106, 3.0000, 3.0000, 3.0000, 3.0172, 3.0000,\n",
      "         3.0000],\n",
      "        [3.0000, 3.0000, 3.0000, 3.1178, 3.0000, 3.1770, 3.0000, 3.1818, 3.0722,\n",
      "         3.0000],\n",
      "        [3.0000, 3.0000, 3.0000, 3.2704, 3.0000, 3.0856, 3.0000, 3.2686, 3.0000,\n",
      "         3.0000],\n",
      "        [3.0000, 3.0000, 3.1334, 3.3345, 3.0000, 3.0000, 3.0000, 3.4957, 3.0000,\n",
      "         3.0000]], grad_fn=<AddBackward0>)\n"
     ]
    }
   ],
   "source": [
    "import torch.nn.functional as F\n",
    "# Define one layer\n",
    "layer = nn.Linear(100, 10)\n",
    "# Define the non-linearity\n",
    "activation = nn.Sigmoid()\n",
    "# Create some random input\n",
    "inputs = torch.rand(32, 100)\n",
    "# inputs.view(16, -1) => 200\n",
    "# inputs.view(16, -1, 4) => 50\n",
    "# Apply our layers\n",
    "outputs = activation(layer(inputs))\n",
    "# Equivalently, as ReLU is parameter-free\n",
    "outputs = F.relu(layer(inputs))\n",
    "y = outputs + 3\n",
    "y.backward()\n",
    "print(outputs)\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now to make this even cleaner, we can define our own `nn.Module`, which is a `Pytorch` class representing models. To do so, we can define a sub-class, and implement the functions `__init__` (defining our layers) and `forward` (explaining how our forward pass will behave)\n",
    "\n",
    "***\n",
    "\n",
    "**Exercise**\n",
    "1. Implement a CNN in Pytorch using the `functional` library\n",
    "\n",
    "***\n",
    "\n",
    "<!--\n",
    "<div class=\"alert alert-info\" markdown=1><h4>Exercise</h4>\n",
    "1. Implement a CNN in Pytorch using the `functional` library\n",
    "</div>\n",
    "-->\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "class CNN(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(CNN, self).__init__()\n",
    "        self.conv1 = nn.Conv2d(3, 6, 5)\n",
    "        \n",
    "        ######################\n",
    "        # YOUR CODE GOES HERE\n",
    "        ######################\n",
    "\n",
    "        ######################\n",
    "        # Solution:\n",
    "        self.pool = nn.MaxPool2d(2, 2)\n",
    "        self.conv2 = nn.Conv2d(6, 16, 5)\n",
    "        self.fc1 = nn.Linear(16 * 5 * 5, 120)\n",
    "        self.fc2 = nn.Linear(120, 84)\n",
    "        self.fc3 = nn.Linear(84, 10)\n",
    "        ######################\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = F.relu(self.conv1(x))\n",
    "        \n",
    "        ######################\n",
    "        # YOUR CODE GOES HERE\n",
    "        ######################\n",
    "\n",
    "        ######################\n",
    "        # Solution:        \n",
    "        x = self.pool(x)\n",
    "        x = self.pool(F.relu(self.conv2(x)))\n",
    "        x = x.view(-1, 16 * 5 * 5)\n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = F.relu(self.fc2(x))\n",
    "        x = self.fc3(x)\n",
    "        ######################\n",
    "        return x\n",
    "\n",
    "model = CNN()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training the network\n",
    "\n",
    "In order to test our CNN, we are going to try to perform image classification. To do so, we can use the simplifications for data loading contained in `torchvision`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torchvision\n",
    "import torchvision.transforms as transforms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `torchvision` package provides pre-coded simplification to download and use the major image datasets, notably `MNIST` and `CIFAR`, which are the baseline datasets for testing image ML models. The output of torchvision datasets are PILImage images of range [0, 1]. We transform them to Tensors of normalized range [-1, 1].\n",
    "\n",
    "In the following code, we are going to load the `CIFAR10` _train_ and _test_ sets. **Note that this code will automatically download the dataset if you did not have it before, and place it in the `data` folder, so this might take a bit of time**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files already downloaded and verified\n",
      "Files already downloaded and verified\n"
     ]
    }
   ],
   "source": [
    "# Transforms to apply to the images\n",
    "transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])\n",
    "# Import the train dataset\n",
    "trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform)\n",
    "trainloader = torch.utils.data.DataLoader(trainset, batch_size=16, shuffle=True, num_workers=2)\n",
    "# Import the test dataset\n",
    "testset = torchvision.datasets.CIFAR10(root='./data', train=False, download=True, transform=transform)\n",
    "testloader = torch.utils.data.DataLoader(testset, batch_size=4, shuffle=False, num_workers=2)\n",
    "# Classes in the CIFAR dataset\n",
    "classes = ('plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "Image : torch.Size([4, 3, 32, 32])\n",
      "batch_labels : ['cat', 'ship', 'ship', 'plane']\n"
     ]
    },
    {
     "ename": "TypeError",
     "evalue": "swapaxes() takes 2 positional arguments but 3 were given",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-59-064cbf33237d>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      6\u001b[0m     \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"batch_labels : {[classes[b] for b in batch_labels]}\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m     \u001b[0mplt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfigure\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfigsize\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m9\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m9\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 8\u001b[0;31m     \u001b[0mplt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mimshow\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbatch_images\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m:\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m:\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m:\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mswapaxes\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m2\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      9\u001b[0m     \u001b[0mplt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshow\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     10\u001b[0m     \u001b[0;32mbreak\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mTypeError\u001b[0m: swapaxes() takes 2 positional arguments but 3 were given"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<Figure size 648x648 with 0 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "for i, (batch_images, batch_labels) in enumerate(testloader):\n",
    "    print(i)\n",
    "    print(f\"Image : {batch_images.shape}\")\n",
    "    print(f\"batch_labels : {[classes[b] for b in batch_labels]}\")\n",
    "    plt.figure(figsize=(9,9))\n",
    "    plt.imshow((batch_images[0, :, :, :].swapaxes(1, 2, 0)))\n",
    "    plt.show()\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using your knowledge from the previous course, you can now define an optimization problem, and implement the training loop for your model\n",
    "\n",
    "***\n",
    "\n",
    "**Exercise**\n",
    "1. Define a `criterion` and `optimizer`\n",
    "2. Fill in the training loop to train your model\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.optim as optim\n",
    "\n",
    "######################\n",
    "# YOUR CODE GOES HERE\n",
    "######################\n",
    "\n",
    "        \n",
    "######################\n",
    "# Solution:\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)\n",
    "######################"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_epochs = 100\n",
    "# Training loop\n",
    "for epoch in range(n_epochs):\n",
    "    running_loss = 0.0\n",
    "    # Go through all batches\n",
    "    for i, data in enumerate(trainloader, 0):\n",
    "        # Get the inputs; data is a list of [inputs, labels]\n",
    "        inputs, labels = data\n",
    "        \n",
    "        ######################\n",
    "        # YOUR CODE GOES HERE\n",
    "        ######################\n",
    "\n",
    "        ######################\n",
    "        # Solution: \n",
    "        # Zero the parameter gradients\n",
    "        optimizer.zero_grad()\n",
    "        # Forward + backward + optimize\n",
    "        outputs = model(inputs)\n",
    "        loss = criterion(outputs, labels)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        ######################\n",
    "        \n",
    "        # Keep track of the loss\n",
    "        running_loss += loss.item()\n",
    "    print(f'loss at epoch {epoch}: {running_loss}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CNN(\n",
      "  (conv1): Conv2d(3, 6, kernel_size=(5, 5), stride=(1, 1))\n",
      "  (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
      "  (conv2): Conv2d(6, 16, kernel_size=(5, 5), stride=(1, 1))\n",
      "  (fc1): Linear(in_features=400, out_features=120, bias=True)\n",
      "  (fc2): Linear(in_features=120, out_features=84, bias=True)\n",
      "  (fc3): Linear(in_features=84, out_features=10, bias=True)\n",
      ")\n",
      "tensor([[ 0.4643,  1.3472, -0.7042, -0.6442, -0.2802, -1.2991, -1.3025, -0.1195,\n",
      "         -0.1503,  1.5305],\n",
      "        [ 0.7539,  2.4305, -1.2714, -0.8580, -0.8926, -1.8478, -1.4893, -0.4473,\n",
      "          0.7394,  2.5573],\n",
      "        [-0.2794, -0.3021,  0.0342,  0.0146,  0.3619, -0.4656, -0.0909,  0.6245,\n",
      "         -1.4118,  0.1575],\n",
      "        [-1.1520, -1.4370,  0.7748,  0.8467,  1.4947, -0.0066,  1.3852,  0.8724,\n",
      "         -2.6392, -1.2578],\n",
      "        [-0.5027, -0.7417, -0.2353,  0.3165,  0.1069, -0.2562, -0.6869,  1.8786,\n",
      "         -1.6387,  0.4327],\n",
      "        [-1.4727, -2.9020,  1.6517,  1.1908,  2.6319,  0.7122,  1.7304,  1.5384,\n",
      "         -3.2966, -2.7199],\n",
      "        [-0.8087, -2.2301,  1.0829,  0.7133,  1.4301,  0.5089, -0.2139,  2.1315,\n",
      "         -3.0065, -1.8095],\n",
      "        [-0.6353, -2.2707,  0.7687,  1.1081,  1.2514,  0.9214, -0.4288,  2.4572,\n",
      "         -2.9334, -2.4289],\n",
      "        [-1.2976, -1.1493,  1.0172,  0.8146,  1.5332,  0.1933,  1.9325,  0.2382,\n",
      "         -2.4728, -1.3650],\n",
      "        [-0.7440, -0.8384,  0.0581,  1.5641, -0.0953,  1.6307, -0.2266,  0.3623,\n",
      "         -1.0119, -1.5647],\n",
      "        [-0.4805, -0.5135, -0.0113,  1.1212,  0.0259,  0.0659, -0.4295,  0.9205,\n",
      "         -2.2968, -0.4320],\n",
      "        [-0.4367,  1.4086, -0.8667,  0.0983, -0.9384, -0.8306, -0.2277, -0.0093,\n",
      "         -0.7568,  1.8025],\n",
      "        [ 1.3857,  0.5201,  0.0380, -0.2156, -0.3007, -1.1159, -2.0112, -0.4127,\n",
      "          0.2257,  0.3484],\n",
      "        [ 1.8040,  2.3176, -1.4531, -1.3646, -1.0467, -1.7188, -3.0968, -0.3802,\n",
      "          2.3983,  2.4642],\n",
      "        [ 3.2519, -1.3321,  1.6562,  0.3272, -0.0530,  0.0699, -2.2741, -1.6921,\n",
      "          1.1684, -3.0744],\n",
      "        [-1.4173, -2.2450,  1.2412,  0.9938,  2.2954,  0.2971,  1.8010,  1.3224,\n",
      "         -3.1574, -2.1454]], grad_fn=<AddmmBackward0>)\n"
     ]
    }
   ],
   "source": [
    "print(model)\n",
    "out = model(inputs)\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that your model is trained, you can test it by feeding some new (unseen) images and see if it is able to classify them correctly"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"layer\"></a>\n",
    "## Coding our own convolutional layer\n",
    "\n",
    "Although `Pytorch` comes packed with pre-implemented layers, we can also very easily define our own layers. This will be useful when you start doing research and propose your own way of processing the information. A large advantage of `Pytorch` is that it performs _automatic gradient differentiation_, this means that we simply have to define how the `forward` pass will work, and `Pytorch` will automatically infer the backpropagation equations, without us having to go through any complicated differentiation\n",
    "\n",
    "In the following, we are going to redefine the `Conv2d` layer, by computing the operation ourselves.\n",
    "\n",
    "***\n",
    "\n",
    "**Exercise**\n",
    "1. Complete the `forward` function to compute a convolution\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyConv2d(nn.Module):\n",
    "    def __init__(self, n_channels, out_channels, kernel_size, dilation=1, padding=0, stride=1):\n",
    "        super(MyConv2d, self).__init__()\n",
    "        self.kernel_size = (kernel_size, kernel_size)\n",
    "        self.kernal_size_number = kernel_size * kernel_size\n",
    "        self.out_channels = out_channels\n",
    "        self.dilation = (dilation, dilation)\n",
    "        self.padding = (padding, padding)\n",
    "        self.stride = (stride, stride)\n",
    "        self.n_channels = n_channels\n",
    "        self.weights = nn.Parameter(torch.Tensor(self.out_channels, self.n_channels, self.kernal_size_number))\n",
    "\n",
    "    def forward(self, x):\n",
    "        width = self.calculateNewWidth(x)\n",
    "        height = self.calculateNewHeight(x)\n",
    "        windows = self.calculateWindows(x)\n",
    "        \n",
    "        ######################\n",
    "        # YOUR CODE GOES HERE\n",
    "        ######################\n",
    "\n",
    "        ######################\n",
    "        # Solution: \n",
    "        result = torch.zeros([x.shape[0] * self.out_channels, width, height], dtype=torch.float32)\n",
    "        for channel in range(x.shape[1]):\n",
    "            for i_convNumber in range(self.out_channels):\n",
    "                xx = torch.matmul(windows[channel], self.weights[i_convNumber][channel]) \n",
    "                xx = xx.view(-1, width, height)\n",
    "                result[i_convNumber * xx.shape[0] : (i_convNumber + 1) * xx.shape[0]] += xx\n",
    "        result = result.view(x.shape[0], self.out_channels, width, height)\n",
    "        ######################\n",
    "        \n",
    "        return result  \n",
    "\n",
    "    def calculateWindows(self, x):\n",
    "        windows = F.unfold(x, kernel_size=self.kernel_size, padding=self.padding, dilation=self.dilation, stride=self.stride)\n",
    "        windows = windows.transpose(1, 2).contiguous().view(-1, x.shape[1], self.kernal_size_number)\n",
    "        windows = windows.transpose(0, 1)\n",
    "        return windows\n",
    "\n",
    "    def calculateNewWidth(self, x):\n",
    "        return ((x.shape[2] + 2 * self.padding[0] - self.dilation[0] * (self.kernel_size[0] - 1) - 1) // self.stride[0]) + 1\n",
    "\n",
    "    def calculateNewHeight(self, x):\n",
    "        return ((x.shape[3] + 2 * self.padding[1] - self.dilation[1] * (self.kernel_size[1] - 1) - 1)// self.stride[1]) + 1\n",
    "\n",
    "# Testing the code directly\n",
    "conv = MyConv2d(3, 1, 3)\n",
    "x = torch.randn(1, 3, 24, 24)\n",
    "out = conv(x)\n",
    "out.mean().backward()\n",
    "# Check that we do have gradients\n",
    "print(conv.weights.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you can use your own `MyConv2d` layer, and use it in real-life scenarios, by trying to change your previous model to use your own layer instead."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Recurrent Neural Networks (RNN)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recurrent neural networks (RNNs) are a family of models designed to process time series and sequential data, which perform remarkably in applications such as speech recognition or machine translation. The ability of RNNs to model correlations between successive computations through recurrent connection make them efficient for temporal problem as they provide a form of _memory_. \n",
    "\n",
    "To model structured sequential data, NNs can be augmented with recurrent loops, which allow to retain information across time steps. Considering a sequence $\\mathbf{X}=\\{\\mathbf{x}_t\\}$, dependencies between elements are managed by having a recurrent hidden state $\\mathbf{h}_t$ at time $t$ in the network. The value of $\\mathbf{h}_t$ at each time depends of the previous time and the input, as depicted in the following figure. \n",
    "\n",
    "<img src=\"images/02_rnn.png\" align=\"center\"/>\n",
    "\n",
    "Formally, each hidden state is updated as\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "\\mathbf{h}_t = \n",
    "\\begin{cases} \n",
    "\\phi_{\\mathbf{\\mathbf{\\theta}}}(\\mathbf{x}_0) & \\mbox{if } t=0 \\\\ \n",
    "\\phi_{\\mathbf{\\mathbf{\\theta}}}(\\mathbf{h}_{t-1},\\mathbf{x}_t), & \\mbox{otherwise} \n",
    "\\end{cases}\n",
    "\\label{eq:RNNhiddenupdate}\n",
    "\\end{equation}\n",
    "$$ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Implementing a simple rnn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's a simple one-to-many vanilla recurrent neural network example in functional form. If we were to produce `h[t]`, we need some weight matrices, `h[t-1]`, `x[t]` and a non-linearity `tanh`.\n",
    "\n",
    "$$\n",
    "h_{t} = tanh(W_{hh}h_{t-1} + W_{xh}x_{t} + B_{h})\n",
    "$$\n",
    "\n",
    "Since this is a **one-to-many** network, we'd want to produce an output `y[t]` at every timestep, thus, we need another weight matrix that accepts a hidden state and project it to an output.\n",
    "\n",
    "$$\n",
    "y_{t} = W_{hy}h_{t} + B_{y}\n",
    "$$\n",
    "\n",
    "Now that we know how to use the `Functional` library of `Pytorch`, we are going to implement our own simple RNN layer as previously. This time, we do not provide the content of the `__init__` function, so think carefully of what parameters you will need and how you need to define them.\n",
    "\n",
    "***\n",
    "\n",
    "**Exercise**\n",
    "1. Complete the `__init__` function to instantiate all required parameters\n",
    "1. Complete the `forward` function to compute the forward pass\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "class RecurrentNetwork(nn.Module):\n",
    "    \n",
    "    def __init__(self, input_size, hidden_size, output_size):\n",
    "        \"\"\" Defines the required variables \"\"\"\n",
    "        super(RecurrentNetwork, self).__init__()\n",
    "        \n",
    "        ######################\n",
    "        # YOUR CODE GOES HERE\n",
    "        ######################\n",
    "\n",
    "        ######################\n",
    "        # Solution: \n",
    "        self.hidden_state = torch.zeros((1, hidden_size))\n",
    "        self.W_hh = nn.Parameter(torch.Tensor(hidden_size, hidden_size))\n",
    "        self.W_xh = nn.Parameter(torch.Tensor(input_size, hidden_size))\n",
    "        self.W_hy = nn.Parameter(torch.Tensor(hidden_size, output_size))\n",
    "        self.Bh = nn.Parameter(torch.Tensor(1, hidden_size))\n",
    "        self.By = nn.Parameter(torch.Tensor(1, output_size))\n",
    "\n",
    "    \n",
    "    def forward(self, x):\n",
    "        \"\"\" Computes the forward pass \"\"\"\n",
    "        \n",
    "        ######################\n",
    "        # YOUR CODE GOES HERE\n",
    "        ######################\n",
    "\n",
    "        ######################\n",
    "        # Solution: \n",
    "        hh = torch.mm(self.hidden_state, self.W_hh)\n",
    "        xh = torch.mm(x, self.W_xh)\n",
    "        self.hidden_state = F.tanh((hh) + (xh) + self.Bh)\n",
    "        hy = torch.mm(self.hidden_state, self.W_hy)\n",
    "        output = hy + self.By\n",
    "        ######################\n",
    "        return output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then forward propagate our information inside our network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "in_size = 4\n",
    "input_vector = torch.ones(16, in_size)\n",
    "simple_network = RecurrentNetwork(in_size, 10, 5)\n",
    "# Notice that same input, but leads to different ouptut at every single time step.\n",
    "print(simple_network(input_vector))\n",
    "print(simple_network(input_vector))\n",
    "print(simple_network(input_vector))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As usual, `Pytorch` also comes packed with some pre-coded recurrent layers. You can go check the documentation to find how to use these."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}