PyTorch_Tutorial_OReilly-empty.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.3"
    },
    "colab": {
      "name": "PyTorch_Tutorial_OReilly.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "sLSxoMS198ZM",
        "xgmsSRw198ZR"
      ],
      "include_colab_link": true
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/dfollereau/oreilly-deeplearning/blob/main/PyTorch_Tutorial_OReilly-empty.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5fP31OuG98Xj"
      },
      "source": [
        "# Introduction to PyTorch - Neural Nets and Beyond\n",
        "\n",
        "## Robert Alvarez - Head of Data Science at Podium Education\n",
        "\n",
        "\n",
        "<!--![](data/img/odsc.png)-->\n",
        "\n",
        "<!---![](data/img/pydata-miami-logo.png)--->\n",
        "\n",
        "<!---![](data/img/GAIC.jpg)--->\n",
        "\n",
        "<!--![](data/img/Robert_Alvarez.jpg)-->\n",
        "\n",
        "![](https://github.com/robert-alvarez/pytorch_tutorial/blob/master/data/img/Oreilly.jpg?raw=1)\n",
        "\n",
        "![](https://github.com/robert-alvarez/pytorch_tutorial/blob/master/data/img/pytorch_logo.png?raw=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C8MMEUpx98Xk"
      },
      "source": [
        "---\n",
        "# Using PyTorch on Google Server\n",
        "\n",
        "Google has made a version of Jupyter Notebook available online for **free** that allow us to use GPUs for faster training time! I do not recommend you use the local installation unless you don't have access to the internet.\n",
        "\n",
        "Go to https://colab.research.google.com and sign in with your Google account. If you do not have a Google account you can create one. From there you can create a new notebook."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FmihBmzK-r04"
      },
      "source": [
        "# PyTorch on Google\n",
        "\n",
        "Google Colab has already pre-installed PyTorch (and many other libraries) so no need to re-install on our virtual machines.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-04-30T21:58:09.320174Z",
          "start_time": "2019-04-30T21:58:06.131991Z"
        },
        "id": "wM7YJvTf98Xl"
      },
      "source": [
        "# usual suspects\n",
        "import os\n",
        "import time\n",
        "import shutil\n",
        "import requests\n",
        "import zipfile\n",
        "from PIL import Image\n",
        "from io import BytesIO\n",
        "\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "from cycler import cycler\n",
        "\n",
        "# the good stuff\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "from torch.autograd import Variable\n",
        "import torch.nn.functional as F\n",
        "from torch.utils.data import Dataset, DataLoader\n",
        "import torchvision\n",
        "from torchvision import datasets, models, transforms\n",
        "from torchsummary import summary\n",
        "\n",
        "# standard sklearn import\n",
        "from sklearn.metrics import accuracy_score\n",
        "\n",
        "# minor changes to plotting functions\n",
        "import matplotlib.pyplot as plt\n",
        "cmap=plt.cm.tab10\n",
        "c = cycler('color', cmap(np.linspace(0,1,10)))\n",
        "plt.rcParams[\"axes.prop_cycle\"] = c\n",
        "\n",
        "\n",
        "# see if GPU is available\n",
        "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
        "\n",
        "%matplotlib inline\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Dm78sdz_v0Qa"
      },
      "source": [
        "# Output should be: device(type='cuda', index=0)\n",
        "# If not, restart runtime with GPU enabled.\n",
        "device"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2XgbYeYa98Xn"
      },
      "source": [
        "# Download all necessary files for the live tutorial\n",
        "\n",
        "data_url = 'https://www.dropbox.com/s/k8ywqfx2hvrh9ny/pytorch_data.zip?dl=1'\n",
        "\n",
        "if not os.path.exists('data'):\n",
        "    # Download the data zip file.\n",
        "    response = requests.get(data_url, stream=True)\n",
        "    zip_path = 'pytorch_data.zip'\n",
        "    with open(zip_path, 'wb') as f:\n",
        "        shutil.copyfileobj(response.raw, f)\n",
        "    # Unzip the file.\n",
        "    with zipfile.ZipFile(zip_path, 'r') as z:\n",
        "        z.extractall()\n",
        "    # Clean up.\n",
        "    os.remove(zip_path)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d9lpGce498Xp"
      },
      "source": [
        "# PyTorch\n",
        "\n",
        "What is PyTorch?\n",
        "\n",
        "* It is a replacement for NumPy to use GPUs\n",
        "* A deep learning platform built for flexibility and speed\n",
        "\n",
        "\n",
        "## Tensor Overview\n",
        "What are tensors?\n",
        "\n",
        "Tensors are similar to NumPy's `ndarrays`\n",
        "\n",
        "We normally think of tensors as a generalization of matrices. In fact, matrices are 2-D tensors!\n",
        "\n",
        "![](https://github.com/robert-alvarez/pytorch_tutorial/blob/master/data/img/grumpy_cat_tensor.jpg?raw=1)\n",
        "\n",
        "Here is a great visualization of tensors from 1-D to 5-D\n",
        "\n",
        "![](https://github.com/robert-alvarez/pytorch_tutorial/blob/master/data/img/tensor_viz.jpeg?raw=1)\n",
        "\n",
        "As mentioned before, since tensors are generalizations of matrices, we should be able to create them in similar ways. We can also expect *most* operations to stay the same. In particular, addition of tensors is the same as for matrices. Multiplication is a bit different, but we won't have to concern ourselves with that in this lecture."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SCw6QKH298Xq"
      },
      "source": [
        "### Tensor Types\n",
        "\n",
        "Torch defines eight CPU tensor types and eight GPU tensor types:\n",
        "\n",
        "| Data type                | dtype                             | CPU Tensor           | GPU Tensor                |\n",
        "|--------------------------|-----------------------------------|----------------------|---------------------------|\n",
        "| 32-bit floating point    | `torch.float32` or `torch.float`  | `torch.FloatTensor`  | `torch.cuda.FloatTensor`  |\n",
        "| 64-bit floating point    | `torch.float64` or `torch.double` | `torch.DoubleTensor` | `torch.cuda.DoubleTensor` |\n",
        "| 16-bit floating point    | `torch.float16` or `torch.half`   | `torch.HalfTensor`   | `torch.cuda.HalfTensor`   |\n",
        "| 8-bit integer (unsigned) | `torch.uint8`                     | `torch.ByteTensor`   | `torch.cuda.ByteTensor`   |\n",
        "| 8-bit integer (signed)   | `torch.int8`                      | `torch.CharTensor`   | `torch.cuda.CharTensor`   |\n",
        "| 16-bit integer (signed)  | `torch.int16` or `torch.short`    | `torch.ShortTensor`  | `torch.cuda.ShortTensor`  |\n",
        "| 32-bit integer (signed)  | `torch.int32` or `torch.int`      | `torch.IntTensor`    | `torch.cuda.IntTensor`    |\n",
        "| 64-bit integer (signed)  | `torch.int64` or `torch.long`     | `torch.LongTensor`   | `torch.cuda.LongTensor`   |\n",
        "\n",
        "**Note**: Tensor types need to match when doing calculations with them."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9b_Zh-XD98Xq"
      },
      "source": [
        "### Numpy ndarrays vs PyTorch Tensors\n",
        "\n",
        "Let's look at the differences between these two common methods of handling arrays. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cDuKQwQo98Xr"
      },
      "source": [
        "# In numpy, we create tensors (arrays) in the following way\n",
        "x1 = np.random.rand(5,3)\n",
        "print(f\"x1 =\\n {x1}\\n\")\n",
        "\n",
        "# Similar to numpy we can create random tensors\n",
        "t1 = torch.rand(5,3, dtype=torch.float64)\n",
        "print(f\"t1 =\\n {t1}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qYXbEFaJ98Xt"
      },
      "source": [
        "# check the type\n",
        "print(f\"t1 is a {type(t1)}\")\n",
        "print(f\"x1 is a {type(x1)}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ckve1bS898Xu"
      },
      "source": [
        "print(f\"t1 is dtype {t1.dtype}\")\n",
        "print(f\"x1 is dtype {x1.dtype}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iC9nKptd98Xw"
      },
      "source": [
        "# create a 3-D tensor in torch and numpy\n",
        "t2 = torch.rand(3, 4, 5)\n",
        "print(f\"t2 =\\n {t2}\\n\")\n",
        "\n",
        "x2 = np.random.rand(3, 4, 5)\n",
        "print(f\"x2 =\\n {x2}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TLLno7I498Xy"
      },
      "source": [
        "long_tensor = torch.zeros(5,3, dtype=torch.long)\n",
        "long_tensor"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WjJxk3x898X1"
      },
      "source": [
        "float_tensor = torch.zeros(5,3, dtype=torch.float64)\n",
        "float_tensor"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "O1zfR0H198X4"
      },
      "source": [
        "long_tensor + float_tensor"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GJXNRaUC98X7"
      },
      "source": [
        "# we can also create explicit tensors\n",
        "x = torch.tensor([2., 3])\n",
        "x, x.type()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2x8zEP4S98X9"
      },
      "source": [
        "# this method creates a new tensor \"y\" that has the same properties (e.g. dtype) as the original tensor \"x\"\n",
        "y = x.new_ones(5,3)\n",
        "print(f'{y.type()}\\n {y}')\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1ldBCZ-Q98X_"
      },
      "source": [
        "# there's a size method as well\n",
        "y.size()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TVYmRi0m98YB"
      },
      "source": [
        "# We can add tensors as well in the usual way you expect\n",
        "x = torch.rand(5,3)\n",
        "x + y"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z56pRLLn98YD"
      },
      "source": [
        "# We can also add like this\n",
        "torch.add(x, y)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S0PQJ4bI98YE"
      },
      "source": [
        "# OR like this!\n",
        "# The `*_` method works just like `inplace=True` in pandas\n",
        "y.add_(x) "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "38U4RzTj98YG"
      },
      "source": [
        "# see!\n",
        "y"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4KD9jmwr98YI"
      },
      "source": [
        "# Indexing/Slicing works just like in numpy\n",
        "y[2, :]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Hfs11ngt98YX"
      },
      "source": [
        "# We can reshape tensors as well though it's called \"view\" in pytorch\n",
        "# support for 'reshape' was added in 2020\n",
        "x = torch.randn(4,4)\n",
        "y = x.view(16)\n",
        "z = x.view(-1, 8)\n",
        "w = x.reshape(2, -1)\n",
        "print(x.size(), y.size(), z.size(), w.size())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XatTg6M498Ya"
      },
      "source": [
        "More on torch operations can be found here: https://pytorch.org/docs/stable/torch.html"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oUce2CSj98Yb"
      },
      "source": [
        "# Converting to/from PyTorch & NumPy is easy\n",
        "a = torch.ones(5)\n",
        "a"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H10V5GL098Ye"
      },
      "source": [
        "# convert to numpy\n",
        "b = a.numpy()\n",
        "b"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uxEOtLfN98Yg"
      },
      "source": [
        "# what do we expect this result to be?\n",
        "a.add_(1)\n",
        "print(a)\n",
        "print(b)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CqnuTdCJ98Yi"
      },
      "source": [
        "# Convert from numpy to torch\n",
        "c = np.random.randn(4,5)\n",
        "print(c)\n",
        "d = torch.from_numpy(c)\n",
        "print(d)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UvnyYAJ698Yk"
      },
      "source": [
        "## Autograd - AKA why PyTorch is awesome\n",
        "\n",
        "Central to all Neural Networks in PyTorch is the `autograd` package. \n",
        "\n",
        "The `autograd` package provides automatic differentiation for all operations on Tensors. It is a define-by-run framework, \n",
        "which means that your backprop is defined by how your code is run, and that every single iteration can be different.\n",
        "\n",
        "Let's look at a basic example of this before turning to Neural Networks."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AqIAiQl798Yk"
      },
      "source": [
        "# Create a tensor and set requires_grad=True to track computation with it\n",
        "x = torch.ones(2,2, requires_grad=True)\n",
        "print(x)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5hCV7jWG98Ym"
      },
      "source": [
        "# Do some operation on said tensor\n",
        "y = 3*x + 7\n",
        "print(y)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OpzudC3v98Yn"
      },
      "source": [
        "# Because y was created as a result of an operation, it now has a grad_fn method\n",
        "y.grad_fn"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xl0fduSM98Yo"
      },
      "source": [
        "# We can do more stuff to y (and thus x) and calculate its derivatives\n",
        "z = 2*y**2\n",
        "w = z.mean()\n",
        "\n",
        "print(z, w)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3Fz1lXOm4LwQ"
      },
      "source": [
        "Let's do that derivative by hand and see that we get the same result.\n",
        "\n",
        "$\\frac{\\partial w}{\\partial x} = \\frac{\\partial w}{\\partial z} \\frac{\\partial z}{\\partial y} \\frac{\\partial y}{\\partial x}$\n",
        "\n",
        "Which gives us \n",
        "\n",
        "$w(x) = \\frac{1}{4} z(x)^2 = \\frac{1}{2} (3x+7)^2$\n",
        "\n",
        "$w'(x) = (3x+7)*3|_{x=1} = 30$"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "U-9pbJF698Yp"
      },
      "source": [
        "# Backpropagation in one line!\n",
        "w.backward()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GrscC7cq98Yr"
      },
      "source": [
        "# et voila!\n",
        "x.grad"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cZBQt0mX98Yt"
      },
      "source": [
        "## Linear Regression Example"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-05-02T00:53:01.316904Z",
          "start_time": "2019-05-02T00:53:01.309295Z"
        },
        "id": "KeTo0Pb_98Yu"
      },
      "source": [
        "n=1000"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-05-02T00:53:01.886074Z",
          "start_time": "2019-05-02T00:53:01.876908Z"
        },
        "id": "hJ4LRmVQ98Yv"
      },
      "source": [
        "x = torch.ones(n,2) \n",
        "x[:,0].uniform_(-1.,1)\n",
        "x[:5]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-05-02T00:55:25.633499Z",
          "start_time": "2019-05-02T00:55:25.617471Z"
        },
        "id": "Yrp1OMEe98Yx"
      },
      "source": [
        "a = torch.tensor([3.,2]); a"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-05-02T00:55:26.210711Z",
          "start_time": "2019-05-02T00:55:26.197229Z"
        },
        "id": "P2k3oBNf98Yy"
      },
      "source": [
        "y = x@a + torch.randn(n)/3"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lPFm3Yy798Y0"
      },
      "source": [
        "plt.scatter(x[:,0], y);"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YGjHe8Hz98Y2"
      },
      "source": [
        "def mse(y, y_pred): \n",
        "    return ((y - y_pred)**2).mean()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BKgWWEro98Y3"
      },
      "source": [
        "a_guess = torch.tensor([-1.,1])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IlVJnrog98Y5"
      },
      "source": [
        "y_hat = x@a_guess # @ is matrix multiplication\n",
        "mse(y_hat, y)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sv6Qha9A98Y7"
      },
      "source": [
        "plt.scatter(x[:,0],y)\n",
        "plt.scatter(x[:,0],y_hat);"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "K4ahptpE98Y8"
      },
      "source": [
        "a_guess = nn.Parameter(a_guess)\n",
        "a_guess"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "75TK6zOR98Y9"
      },
      "source": [
        "def update():\n",
        "    '''\n",
        "    function to update tensor using SGD\n",
        "    '''\n",
        "    y_hat = x@a_guess\n",
        "    loss = mse(y, y_hat)\n",
        "    if t % 10 == 0: print(loss)\n",
        "    loss.backward()\n",
        "    with torch.no_grad():\n",
        "        a_guess.sub_(lr * a_guess.grad)\n",
        "        a_guess.grad.zero_()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ce9_sd1v98Y-"
      },
      "source": [
        "lr = 1e-1\n",
        "for t in range(100): \n",
        "    update()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DeUXfjdV98ZB"
      },
      "source": [
        "plt.scatter(x[:,0], y)\n",
        "plt.scatter(x[:,0], x@a_guess.detach()); # detach() removes gradient from `a` so it can be plotted. otherwise matplotlib gives error"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FSjr7WCF98ZG"
      },
      "source": [
        "### Normal Equations\n",
        "Although we solved for the line of best fit using stochastic gradient descent,\n",
        "we could have also used the [Normal Equations](https://en.wikipedia.org/wiki/Linear_least_squares#Derivation_of_the_normal_equations) given by\n",
        "\n",
        "$$ \\theta = (X^T X)^{-1} X^T \\vec y$$\n",
        "\n",
        "where $\\theta$ is the vector we called `a` in the linear regression code above"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-05-02T00:57:18.751491Z",
          "start_time": "2019-05-02T00:57:18.743154Z"
        },
        "id": "G9Qwa5o498ZG"
      },
      "source": [
        "# Exercise:\n",
        "# Use pytorch to solve for theta using the normal equations\n",
        "# plot your result when you are finished\n",
        "\n",
        "# HINT:\n",
        "# matrix inverse in pytorch: x.inverse()\n",
        "# matrix transpose in pytorch: torch.t(x)\n",
        "\n",
        "# Code goes here"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XhcElduD98ZH"
      },
      "source": [
        "---\n",
        "# Neural Nets"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IUSiq8Hs98ZI"
      },
      "source": [
        "Neural networks (NNs) are special forms of nonlinear regressions where the decision system for which the NN is built mimics the way \n",
        "the brain is supposed to work (whether it works like a NN is up for grabs, of course).\n",
        "\n",
        "Like many of the algorithms we have seen in classical machine learning, it is a supervised learning technique that can perform complex tasks."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nQ814fgi98ZI"
      },
      "source": [
        "## Perceptrons"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5YQwp2rp98ZJ"
      },
      "source": [
        "The basic building block of a neural network is a perceptron. A perceptron is like a neuron in a human brain. It takes inputs \n",
        "(e.g. sensory in a real brain) and then produces an output signal. An entire network of perceptrons is called a neural net.\n",
        "\n",
        "![linear regression](https://github.com/robert-alvarez/pytorch_tutorial/blob/master/data/img/NN_perceptron.png?raw=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "00sR2ipX98ZJ"
      },
      "source": [
        "In general, a perceptron could have more or fewer inputs."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XldQsuBe98ZK"
      },
      "source": [
        "Instead of assigning equal weight to each of the inputs, we can assign real numbers $w_1, w_2, \\ldots$ expressing the importance of the respective inputs to the output. The nueron's output, 0 or 1, is determined whether the weighted sum $\\sum_j w_j x_j$ is less than or greater than some *threshold value*.\n",
        "\n",
        "Perceptrons may emit continuous signals or binary $(0,1)$ signals. In the case of a credit card application, the final perceptron is a binary one (approved or denied). Such perceptrons are implemented by means of squashing functions. For example, a really simple squashing function is one that issues a 1 if the function value is positive and a $-1$ if it is negative."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tb28h7Ll98ZK"
      },
      "source": [
        "To put this in more mathematical terms, let $z = \\sum_{j=0}^n w_j x_j$ . \n",
        "Then the *activation function* $\\phi(z)$ is defined as \n",
        "\n",
        "$$\n",
        "\\phi(z) =\n",
        "\\begin{cases}\n",
        "-1 & \\text{if } z < \\theta\\\\\n",
        " 1 & \\text{if } z \\geq \\theta\n",
        "\\end{cases}\n",
        "$$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "if9rqZt-98ZL"
      },
      "source": [
        "![](https://github.com/robert-alvarez/pytorch_tutorial/blob/master/data/img/activation-function.png?raw=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b5JMM3rJ98ZL"
      },
      "source": [
        "The whole point of the perceptron is to mimic how a single nueron in the brain works: it either *fires* or it doesn't. Thus, the \n",
        "perceptron rule is fairly simple and can be summarized by the following steps.\n",
        "​\n",
        "* Initialize the weights to zero or small random numbers\n",
        "* For each training sample $\\textbf{x}_n$ perform the following steps:\n",
        "    * Compute the output value $y$\n",
        "    * Calculate error in $y$ vs $\\hat y$\n",
        "    * Update the weights\n",
        "  \n",
        "Here, the output value is the class label predicted by the activation function that we defined earlier, and the\n",
        "simultaneous update of weight $w_j$ in the weight vector $\\textbf{w}$ can be more formally written as\n",
        "​\n",
        "$$\\bar w_j = w_j + \\Delta w_j$$\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sLSxoMS198ZM"
      },
      "source": [
        "## Fitting a model\n",
        "\n",
        "Let's go back to our linear perceptron. It has the following parameters:\n",
        "\n",
        "* $x_i$: inputs\n",
        "* $y$ : output\n",
        "* $w_i$: learned weights\n",
        "\n",
        "What we would like to do is adjust the $w_i$'s until our model has the best fit.\n",
        "\n",
        "First, initialize the $w_i$'s in some meaningful way (usually they're drawn from a randon uniform distribution).\n",
        "\n",
        "Then, we put it into the usual **algorithm workflow:**\n",
        "\n",
        "* calculate prediction $\\hat y$\n",
        "* calculate Loss function $L(y, \\hat y)$\n",
        "* update weights using backpropagation\n",
        "\n",
        "### Loss function and backpropagation\n",
        "\n",
        "To figure out how well our prediction was during each epoch, we'll use a basic loss function, mean squared error (MSE):\n",
        "\n",
        "$L(y,\\hat{y}) = ||~ y-\\hat{y} ~||^2$,\n",
        "\n",
        "ultimately trying to find $L_{\\rm min}$, defined by the point in parameter space where $\\nabla_{w_i} L = 0$.\n",
        "\n",
        "Per-iteration update: \n",
        "\n",
        "$ w_i \\to w_i - \\eta \\nabla_{w_i} L $,\n",
        "\n",
        "where $\\eta$ is known as the learning rate; too small and takes very long to converge, too big and you oscillate about the minimum.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kd5YUrwu98ZM"
      },
      "source": [
        "## A basic example\n",
        "​\n",
        "We'll build a Rube Goldberg adding machine that will illustrate how neural nets work."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7YHrn_ov98ZM"
      },
      "source": [
        "model = nn.Sequential(\n",
        "    nn.Linear(2, 1, bias=False))\n",
        "\n",
        "optimizer = torch.optim.Adam(model.parameters())\n",
        "criterion = nn.MSELoss()\n",
        "print(model)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LJrYvWGK98ZO"
      },
      "source": [
        "total_loss = []\n",
        "num_samples = 10_000\n",
        "\n",
        "for num in range(1, num_samples+1):\n",
        "    # Progress bar indicator\n",
        "    if num % (num_samples//5) == 0:\n",
        "        print('{0}: %: {1:.3f}'.format(num,num/num_samples * 100))\n",
        "        \n",
        "    # data prep\n",
        "    x = 4*torch.rand(2) #generate two random numbers uniformly on (0,4)\n",
        "    data, target = Variable(x), Variable(x[0] + x[1])\n",
        "    \n",
        "    # Feed forward through NN\n",
        "    output = model(data)\n",
        "    loss = criterion(output, target)\n",
        "    total_loss.append(loss)\n",
        "    \n",
        "    # Backpropagation\n",
        "    optimizer.zero_grad()\n",
        "    loss.backward()\n",
        "    optimizer.step()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xOtHnPe598ZP"
      },
      "source": [
        "fig,ax=plt.subplots(figsize=(11,8))\n",
        "ax.plot(total_loss,marker='.',ls='',markersize=1.)\n",
        "ax.set_ylim(0,);ax.set_xlim(0,);ax.grid(alpha=0.2);\n",
        "ax.set_xlabel('training examples');ax.set_ylabel('mean squared loss');"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZPNZtx5598ZQ"
      },
      "source": [
        "x,y=np.linspace(-5,8,100),[]\n",
        "\n",
        "for xx in x:\n",
        "    yy=model(torch.tensor([xx,2], dtype=torch.float32)).data.cpu().numpy()[0]\n",
        "    y.append(yy)\n",
        "y=np.array(y)\n",
        "\n",
        "fig,ax=plt.subplots(figsize=(11,8))\n",
        "ax.plot([-5,8],[-3,10],lw=2.0,label='actual',alpha=0.7)\n",
        "ax.fill_betweenx([-3,10],0,4,alpha=0.2)\n",
        "ax.scatter(x,y,marker='.',s=3.,label='prediction',color='r')\n",
        "ax.text(0.2,0,'Where we have \\ntraining data')\n",
        "ax.legend()\n",
        "ax.set_ylim(-3,10);ax.set_xlim(-5,8);\n",
        "ax.grid(alpha=0.2);\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xgmsSRw198ZR"
      },
      "source": [
        "## Feedforward Neural Network\n",
        "\n",
        "![General Feed Forward Network](https://github.com/robert-alvarez/pytorch_tutorial/blob/master/data/img/neural_net_123.png?raw=1)\n",
        "\n",
        "For our case of learning linear relationships, the modification to the linear regression architecture is depicted below:\n",
        "\n",
        "\n",
        "![non-linear activation](https://github.com/robert-alvarez/pytorch_tutorial/blob/master/data/img/linear_regression_activation.png?raw=1)\n",
        "\n",
        "where\n",
        "\n",
        "$$\\varphi(z) = \\frac{1}{1+e^{-z}}$$\n",
        "\n",
        "is the so-called sigmoid function; this is typically the activation function that is first introduced, I think because of historical reasons. In modern practice, it finds most of its use in transforming single outputs from a NN into a probability. It's worth noting that if your NN will output multiple probabilities, for example, if your NN will categorize between black cats, red cats, white cats, etc., a multi-dimensional generalization of the sigmoid, called the softmax function, is typically used. \n",
        "\n",
        "The motivation behind adding an activation function is the hope that the NN model may capture non-linear relationships that exist in the data. Below are some commonly used activation functions. \n",
        "\n",
        "![activation functions](https://github.com/robert-alvarez/pytorch_tutorial/blob/master/data/img/activation_function_list.png?raw=1)\n",
        "\n",
        "In practice, a lot of architectures use the rectified linear unit (ReLU), along with it's close cousin, the so-called leaky-ReLU. In introducing this idea though, we'll focus on the sigmoid which maps real numbers from $(-\\infty,\\infty) \\to [0,1]$.\n",
        "\n",
        "Of course our data is linear in the case of a straight line (!) but let's see what happens if we try to force a non-linear activation layer to capture a linear relationship.."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tcvWinIf98ZR"
      },
      "source": [
        "## Non-linear model for a linear relationship\n",
        "\n",
        "### Deep Feedforward Network with sigmoid activation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-05-02T00:26:19.752876Z",
          "start_time": "2019-05-02T00:26:19.742303Z"
        },
        "id": "r4Qatq-r98ZS"
      },
      "source": [
        "model = nn.Sequential(\n",
        "            nn.Linear(2, 20),\n",
        "            nn.Sigmoid(),\n",
        "            nn.Linear(20, 20),\n",
        "            nn.Sigmoid(),\n",
        "            nn.Linear(20, 20),\n",
        "            nn.Sigmoid(),\n",
        "            nn.Linear(20, 1))\n",
        "\n",
        "optimizer = torch.optim.Adam(model.parameters())\n",
        "criterion = nn.MSELoss()\n",
        "print(model)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-05-02T00:47:02.232283Z",
          "start_time": "2019-05-02T00:46:52.792621Z"
        },
        "id": "LJV3Gs4Z98ZT"
      },
      "source": [
        "total_loss = []\n",
        "num_samples = 10_000\n",
        "\n",
        "for num in range(1, num_samples+1):\n",
        "    # Progress bar indicator\n",
        "    if num % (num_samples//5) == 0:\n",
        "        print('{0}: %: {1:.3f}'.format(num,num/num_samples * 100))\n",
        "        \n",
        "    # data prep\n",
        "    x = 4*torch.rand(2) #generate two random numbers uniformly on (0,4)\n",
        "    data, target = Variable(x), Variable(x[0] + x[1])\n",
        "    \n",
        "    # Feed forward through NN\n",
        "    output = model(data)\n",
        "    loss = criterion(output, target)\n",
        "    total_loss.append(loss)\n",
        "    \n",
        "    # Backpropagation\n",
        "    optimizer.zero_grad()\n",
        "    loss.backward()\n",
        "    optimizer.step()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-05-02T00:47:24.128293Z",
          "start_time": "2019-05-02T00:47:23.259386Z"
        },
        "id": "sg8t7Nju98ZU"
      },
      "source": [
        "fig,ax=plt.subplots(figsize=(11,8))\n",
        "ax.plot(total_loss,marker='.',ls='',markersize=.8)\n",
        "ax.set_ylim(0,);ax.set_xlim(0,);ax.grid(alpha=0.2);\n",
        "ax.set_xlabel('training examples');ax.set_ylabel('mean squared loss');\n",
        "\n",
        "x,y=np.linspace(-5,8,100),[]\n",
        "\n",
        "for xx in x:\n",
        "    yy=model(torch.tensor([xx,2],dtype=torch.float32)).data.cpu().numpy()[0]\n",
        "    y.append(yy)\n",
        "y=np.array(y)\n",
        "\n",
        "fig,ax=plt.subplots(figsize=(11,8))\n",
        "ax.plot([-5,8],[-3,10],lw=2.0,label='actual',alpha=0.7)\n",
        "ax.fill_betweenx([-3,10],0,4,alpha=0.2)\n",
        "ax.scatter(x,y,marker='.',s=3.,label='prediction',color='r')\n",
        "ax.text(0.2,0,'Where we have \\ntraining data')\n",
        "ax.legend()\n",
        "ax.set_ylim(-3,10);ax.set_xlim(-5,8);\n",
        "ax.grid(alpha=0.2);\n",
        "ax.set_title('Sigmoid activation function');"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8MF1JD0898ZV"
      },
      "source": [
        "## ReLU activation function"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OkrdWaTZ98ZX"
      },
      "source": [
        "model = nn.Sequential(\n",
        "            nn.Linear(2, 20),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(20, 20),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(20, 20),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(20, 1))\n",
        "\n",
        "optimizer = torch.optim.Adam(model.parameters())\n",
        "criterion = nn.MSELoss()\n",
        "print(model)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "81jF1PRV98ZY"
      },
      "source": [
        "total_loss = []\n",
        "num_samples = 10_000\n",
        "\n",
        "for num in range(1, num_samples+1):\n",
        "    # Progress bar indicator\n",
        "    if num % (num_samples//5) == 0:\n",
        "        print('{0}: %: {1:.3f}'.format(num,num/num_samples * 100))\n",
        "        \n",
        "    # data prep\n",
        "    x = 4*torch.rand(2) #generate two random numbers uniformly on (0,4)\n",
        "    data, target = Variable(x), Variable(x[0] + x[1])\n",
        "    \n",
        "    # Feed forward through NN\n",
        "    output = model(data)\n",
        "    loss = criterion(output, target)\n",
        "    total_loss.append(loss)\n",
        "    \n",
        "    # Backpropagation\n",
        "    optimizer.zero_grad()\n",
        "    loss.backward()\n",
        "    optimizer.step()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_8lVbhVS98ZZ"
      },
      "source": [
        "fig,ax=plt.subplots(figsize=(11,8))\n",
        "ax.plot(total_loss,marker='.',ls='',markersize=.8)\n",
        "ax.set_ylim(0,);ax.set_xlim(0,);ax.grid(alpha=0.2);\n",
        "ax.set_xlabel('training examples');ax.set_ylabel('mean squared loss');\n",
        "\n",
        "x,y=np.linspace(-5,8,100),[]\n",
        "\n",
        "for xx in x:\n",
        "    yy=model(torch.tensor([xx,2],dtype=torch.float32)).data.cpu().numpy()[0]\n",
        "    y.append(yy)\n",
        "y=np.array(y)\n",
        "\n",
        "fig,ax=plt.subplots(figsize=(11,8))\n",
        "ax.plot([-5,8],[-3,10],lw=2.0,label='actual',alpha=0.7)\n",
        "ax.fill_betweenx([-3,10],0,4,alpha=0.2)\n",
        "ax.scatter(x,y,marker='.',s=3.,label='prediction',color='r')\n",
        "ax.text(0.2,0,'Where we have \\ntraining data')\n",
        "ax.legend()\n",
        "ax.set_ylim(-3,10);ax.set_xlim(-5,8);\n",
        "ax.grid(alpha=0.2);\n",
        "ax.set_title('ReLu activation function');"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ge84bd3z98Za"
      },
      "source": [
        "## Teaching a machine to draw circles\n",
        "\n",
        "Here the NN learns attempts to learn the 2d rotation matrix, parameterized by the generator of rotations in two dimensions:\n",
        "\n",
        "$R={\\begin{bmatrix}\\cos \\theta &-\\sin \\theta \\\\\\sin \\theta &\\cos \\theta \\\\\\end{bmatrix}}$"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-04-30T05:42:28.228388Z",
          "start_time": "2019-04-30T05:42:28.187133Z"
        },
        "id": "D-LHtrI998Zb"
      },
      "source": [
        "# First some helper functions for plotting and model training\n",
        "def train_models(models, optimizers, num_samples=1000, circle_interval=1.0, save_models=False, cuda=torch.cuda.is_available(), recurrent=False):\n",
        "    total_loss = []\n",
        "    for num in range(1, num_samples+1):\n",
        "        # progress indicator \n",
        "        if num % (num_samples//20) ==0:\n",
        "            print('{0}: %: {1:.3f}'.format(num, num/num_samples * 100))\n",
        "            \n",
        "        # data calc \n",
        "        # take a random point on the circle of radius 1\n",
        "        x, theta = torch.ones(2), circle_interval*2*np.pi*torch.rand(1)\n",
        "        R = torch.zeros(2,2)\n",
        "        R[0,:] = torch.Tensor([np.cos(theta[0]),-np.sin(theta[0])])\n",
        "        R[1,:] = torch.Tensor([np.sin(theta[0]), np.cos(theta[0])])\n",
        "        \n",
        "        data, target = Variable(theta), Variable(torch.mv(R,x))\n",
        "        \n",
        "        # Check if GPU can be used\n",
        "        if cuda:\n",
        "            data, target = data.cuda(), target.cuda()\n",
        "            \n",
        "        # learning phases \n",
        "        for idx, model in enumerate(models):\n",
        "            loss_iter = []\n",
        "            # forward \n",
        "            if recurrent:\n",
        "                output = model(data,None)\n",
        "            else:\n",
        "                output = model(data)\n",
        "            loss = criterion(output, target)\n",
        "            loss_iter.append(loss.data.item())\n",
        "            # backward \n",
        "            optimizers[idx].zero_grad()\n",
        "            loss.backward()\n",
        "            optimizers[idx].step()\n",
        "        total_loss.append(np.mean(loss_iter))\n",
        "        \n",
        "    # save model state \n",
        "    if save_models:\n",
        "        for l,model in enumerate(models):\n",
        "            torch.save(model.state_dict(), 'rotations_{}.pth'.format(l))\n",
        "    return total_loss,theta\n",
        "        \n",
        "def plot_circles(models, offset=0, CI=False):\n",
        "    fig, axes = plt.subplots(figsize=(5*3,3.9),ncols=3)\n",
        "    x = torch.ones(2)\n",
        "    for k,ax in enumerate(axes):\n",
        "        ax.scatter(x[0],x[1], facecolors='none', edgecolors='r')\n",
        "        ax.scatter(x[0],x[1], facecolors='none', edgecolors='b')\n",
        "        x_real, y_real = [],[]\n",
        "        x_mean, y_mean = [],[]\n",
        "        x_std, y_std = [],[]\n",
        "        for theta in np.linspace((k+offset) *2*np.pi,(k+1+offset) *2*np.pi,300):\n",
        "            x_model,y_model = [],[]\n",
        "            # synthetic (real) data \n",
        "            data = Variable(torch.Tensor([theta]))#.cuda()\n",
        "            R = torch.zeros(2,2)\n",
        "            R[0,:] = torch.Tensor([np.cos(theta),-np.sin(theta)])\n",
        "            R[1,:] = torch.Tensor([np.sin(theta), np.cos(theta)])\n",
        "            real = torch.mv(R,x)\n",
        "            x_real.append(real[0].numpy())\n",
        "            y_real.append(real[1].numpy())\n",
        "            # predict w/ all models \n",
        "            for model in models:\n",
        "                if torch.cuda.is_available():\n",
        "                    model.cpu()\n",
        "                    outputs=model(data).data\n",
        "                    xx_model, yy_model = outputs[0],outputs[1]\n",
        "                    x_model.append(xx_model.numpy())\n",
        "                    y_model.append(yy_model.numpy())\n",
        "                else:\n",
        "                    outputs=model(data).data\n",
        "                    xx_model, yy_model = outputs[0],outputs[1]\n",
        "                    x_model.append(xx_model.numpy())\n",
        "                    y_model.append(yy_model.numpy())\n",
        "            # summarize all model predictions \n",
        "            x_mean.append(np.mean(x_model))\n",
        "            y_mean.append(np.mean(y_model))\n",
        "            x_std.append(np.std(x_model))\n",
        "            y_std.append(np.std(y_model))\n",
        "        # plotting data \n",
        "        ax.scatter(x_real,y_real, facecolors='none', edgecolors='r',label='real data',s=2.)\n",
        "        ax.scatter(x_mean,y_mean, facecolors='none', edgecolors='k',label='model data', alpha=0.9,s=2.)\n",
        "        if CI:\n",
        "            ax.fill_betweenx(y_mean,x_mean-3*np.array(x_std),x_mean+3*np.array(x_std), alpha=0.1,color='b')\n",
        "            ax.fill_between(x_mean,y_mean-3*np.array(y_std),y_mean+3*np.array(y_std), alpha=0.1,color='b')\n",
        "        ax.legend()\n",
        "        ax.set_ylim(-2,2);ax.set_xlim(-2,2);ax.grid(alpha=0.3)\n",
        "        ax.set_title(r'${}\\pi \\leq \\theta \\leq {}\\pi$'.format(2*(k+offset),2*(k+1+offset)),y=1.01);\n",
        "    \n",
        "    return x_mean, y_mean, np.array(x_std), np.array(y_std)\n",
        "        \n",
        "def weight_init(m): # so-called xavier normalization https://arxiv.org/abs/1211.5063\n",
        "    if isinstance(m, nn.Linear):\n",
        "        size = m.weight.size()\n",
        "        fan_out = size[0]\n",
        "        fan_in = size[1]\n",
        "        variance = np.sqrt(2.0/(fan_in + fan_out))\n",
        "        m.weight.data.normal_(0.0, variance)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-04-30T05:47:50.984630Z",
          "start_time": "2019-04-30T05:45:17.951543Z"
        },
        "id": "aLPdSOBI98Zd"
      },
      "source": [
        "num_nodes=10\n",
        "\n",
        "# Usually we define neural nets as a class in pytorch\n",
        "class Rotations(nn.Module):\n",
        "    def __init__(self):\n",
        "        super(Rotations, self).__init__()\n",
        "        self.layer1 = nn.Sequential(\n",
        "            nn.Linear(1,num_nodes),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(num_nodes,2))      \n",
        "    def forward(self, x):\n",
        "        out=self.layer1(x)\n",
        "        return out\n",
        "\n",
        "model = Rotations().to(device)\n",
        "optimizer = torch.optim.Adam(model.parameters())\n",
        "criterion = nn.MSELoss()\n",
        "print(model)\n",
        "\n",
        "total_loss, theta = train_models([model], [optimizer], num_samples=100000, circle_interval=2.0)\n",
        "\n",
        "fig,ax = plt.subplots(figsize=(11,8))\n",
        "ax.plot(total_loss,marker='.',ls='',markersize=.8)\n",
        "ax.set_ylim(0,);ax.set_xlim(0,);ax.grid(alpha=0.2);\n",
        "ax.set_xlabel('training examples');ax.set_ylabel('mean squared loss');\n",
        "\n",
        "output=plot_circles([model],offset=0,CI=False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vXBgBf1O98Zg"
      },
      "source": [
        "### Will a deeper network perform better?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-04-30T06:05:29.936212Z",
          "start_time": "2019-04-30T06:00:59.778558Z"
        },
        "id": "Gk0GC2Ry98Zg"
      },
      "source": [
        "# Add more layers to the model and see if performance on the test set increases\n",
        "num_nodes=10\n",
        "class Rotations(nn.Module):\n",
        "    def __init__(self):\n",
        "        super(Rotations, self).__init__()\n",
        "        self.layer1 = nn.Sequential(\n",
        "            nn.Linear(1,num_nodes),\n",
        "            nn.ELU(),\n",
        "            nn.Linear(num_nodes,num_nodes),\n",
        "            nn.ELU(),\n",
        "            nn.Linear(num_nodes,num_nodes),\n",
        "            nn.ELU(),\n",
        "            nn.Linear(num_nodes,2))      \n",
        "    def forward(self, x):\n",
        "        out=self.layer1(x)\n",
        "        return out\n",
        "\n",
        "model=Rotations().to(device)\n",
        "optimizer=torch.optim.Adam(model.parameters())\n",
        "criterion=nn.MSELoss()\n",
        "print(model)\n",
        "\n",
        "total_loss,theta=train_models([model],[optimizer],num_samples=100000,circle_interval=2.0)\n",
        "\n",
        "fig,ax=plt.subplots(figsize=(7,5))\n",
        "ax.plot(total_loss,marker='.',ls='',markersize=.8)\n",
        "ax.set_ylim(0,);ax.set_xlim(0,);ax.grid(alpha=0.2);\n",
        "ax.set_xlabel('training examples');ax.set_ylabel('mean squared loss');\n",
        "\n",
        "output=plot_circles([model],offset=0,CI=False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qajejnfa98Zh"
      },
      "source": [
        "## Parallel approach"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AABjjzF098Zh"
      },
      "source": [
        "One simple approach to apply the method described by Yarin Gal, is to take multiple of the same models and train them independently. This allows each model to take independent paths through parameter space, usually finding their way near some optimal minima. In practice, this allows you to hedge the risk of getting stuck in some local minima and missing out on the global one, if it exists. \n",
        "\n",
        "A visual way of understanding the situation of training a machine learning model, in general, is by considering a 3D surface plot where the x and y dimensions are two parameters you may modify, with the loss on the z axis, or height, which your aim is to minimize. \n",
        "\n",
        "![landscape of deep learning models](https://github.com/robert-alvarez/pytorch_tutorial/blob/master/data/img/graphical-idea-backprop.png?raw=1)\n",
        "\n",
        "The surface that the data carves out in this space is predicated by the data; the aim of the model design is then to build a model flexible and robust enough to find the global minima, but not overly complex enough to overfit and get stuck at a local minima. Also, if your model is too simple, it can skip right over all the minima altogether, and not learn the nuance of the process described by the data. Having too high/low of a learning rate can also make the training process difficult."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gsWWfgQY98Zi"
      },
      "source": [
        "---\n",
        "# Image Recognition and Transfer Learning\n",
        "\n",
        "![Convolutional Architecture](https://github.com/robert-alvarez/pytorch_tutorial/blob/master/data/img/cnn_architecture.png?raw=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J8r6k9iz98Zi"
      },
      "source": [
        "# Transfer Learning\n",
        "\n",
        "Transfer learning is one of the most useful discoveries to come out of the computer vision community. Stated simply, transfer learning allows one model that was trained on different types of images, e.g. dogs vs cats, to be used for a different set of images, e.g. planes vs trains, while reducing the training time dramatically. When Google released ImageNet, they stated it took them over 14 **days** to train the model on some of the most powerful GPUs available at the time. Now, with transfer learning, we will train an, albeit smaller, model in less than 5 minutes.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EF9D_vi898Zi"
      },
      "source": [
        "The philosophy behind transfer learning is simple. We keep the \"base\" layers of a model frozen since the weights have already been tuned to identify patterns such as lines, circles, and other shapes, and insert layers at the end that will be tuned for the specific task at hand.\n",
        "\n",
        "![](https://github.com/robert-alvarez/pytorch_tutorial/blob/master/data/img/NN_vision.png?raw=1)\n",
        "\n",
        "![](https://github.com/robert-alvarez/pytorch_tutorial/blob/master/data/img/transfer_learning.png?raw=1)\n",
        "\n",
        "\n",
        "For our task, let's take a look at the King and Queen of the Miami food scene: \n",
        "\n",
        "**The Cuban Sandwich**  \n",
        "\n",
        "![](https://github.com/robert-alvarez/pytorch_tutorial/blob/master/data/img/cubano.jpg?raw=1)\n",
        "\n",
        "**The Stuffed Arepa**\n",
        "\n",
        "![](https://github.com/robert-alvarez/pytorch_tutorial/blob/master/data/img/arepa.jpg?raw=1)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-04-29T13:46:28.719471Z",
          "start_time": "2019-04-29T13:46:28.704878Z"
        },
        "id": "HHazrBAH98Zj"
      },
      "source": [
        "normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],\n",
        "                                 std=[0.229, 0.224, 0.225])\n",
        " \n",
        "data_transforms = {\n",
        "    'train':\n",
        "        transforms.Compose([\n",
        "            transforms.Resize((224,224)),\n",
        "            transforms.RandomAffine(0, shear=10, scale=(0.8,1.2)),\n",
        "            transforms.RandomHorizontalFlip(),\n",
        "            transforms.ToTensor()]),\n",
        "    'validation':\n",
        "        transforms.Compose([\n",
        "            transforms.Resize((224,224)),\n",
        "            transforms.ToTensor()])}\n",
        " \n",
        "image_datasets = {\n",
        "    'train':\n",
        "        datasets.ImageFolder('data/cva_data/train', data_transforms['train']),\n",
        "    'validation':\n",
        "        datasets.ImageFolder('data/cva_data/validation', data_transforms['validation'])}\n",
        " \n",
        "dataloaders = {\n",
        "    'train':\n",
        "        torch.utils.data.DataLoader(\n",
        "            image_datasets['train'],\n",
        "            batch_size=32,\n",
        "            shuffle=True,\n",
        "            num_workers=2),\n",
        "    'validation':\n",
        "        torch.utils.data.DataLoader(\n",
        "            image_datasets['validation'],\n",
        "            batch_size=32,\n",
        "            shuffle=False,\n",
        "            num_workers=2)}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-05-02T01:28:02.580919Z",
          "start_time": "2019-05-02T01:28:01.614605Z"
        },
        "id": "aYDftGNq98Zk"
      },
      "source": [
        "model = models.resnet50(pretrained=True).to(device)\n",
        " \n",
        "# freeze the weights\n",
        "for param in model.parameters():\n",
        "    param.requires_grad = False\n",
        "    \n",
        "# modify the final layer of resnet50 (called 'fc')\n",
        "# originally model.fc = nn.Linear(2048, 1000) for the 1000 image classes\n",
        "# we modify it to our specific needs\n",
        "\n",
        "model.fc = nn.Sequential(\n",
        "    nn.Linear(2048, 128),\n",
        "    # inplace=True is a shortcut to not modify the forward method \n",
        "    nn.ReLU(inplace=True),\n",
        "    nn.Linear(128, 2)).to(device)\n",
        " \n",
        "criterion = nn.CrossEntropyLoss()\n",
        "optimizer = torch.optim.Adam(model.fc.parameters())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3IikbBD698Zl"
      },
      "source": [
        "# Get estimate of model size: number of parameters, storage, etc.\n",
        "\n",
        "# input size is based off of the dimensions of a single example in our dataset.\n",
        "summary(model, input_size=(3, 224, 224))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Uvz_TM4_98Zm"
      },
      "source": [
        "def train_model(model, criterion, optimizer, num_epochs=3):\n",
        "    for epoch in range(num_epochs):\n",
        "        print(f'Epoch {epoch + 1}/{num_epochs}')\n",
        "        print('-' * 10)\n",
        "\n",
        "        for phase in ['train', 'validation']:\n",
        "            if phase == 'train':\n",
        "                model.train()\n",
        "            else:\n",
        "                model.eval()\n",
        " \n",
        "            running_loss = 0.0\n",
        "            running_corrects = 0\n",
        " \n",
        "            for inputs, labels in dataloaders[phase]:\n",
        "                inputs = inputs.to(device)\n",
        "                labels = labels.to(device)\n",
        " \n",
        "                outputs = model(inputs)\n",
        "                loss = criterion(outputs, labels)\n",
        " \n",
        "                if phase == 'train':\n",
        "                    optimizer.zero_grad()\n",
        "                    loss.backward()\n",
        "                    optimizer.step()\n",
        " \n",
        "                _, preds = torch.max(outputs, 1)\n",
        "                running_loss += loss.item() * inputs.size(0)\n",
        "                running_corrects += torch.sum(preds == labels.data)\n",
        " \n",
        "            epoch_loss = running_loss / len(image_datasets[phase])\n",
        "            epoch_acc = running_corrects.double() / len(image_datasets[phase])\n",
        " \n",
        "            print(f'{phase} loss: {epoch_loss}, acc: {epoch_acc}')\n",
        "    return model\n",
        " \n",
        "model_trained = train_model(model, criterion, optimizer, num_epochs=3)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HG3txGvM98Zn"
      },
      "source": [
        "### Saving Models"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZStqUrY998Zo"
      },
      "source": [
        "torch.save(model_trained.state_dict(),'models/cva_weights_new.pt')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dbmTmF8m98Zq"
      },
      "source": [
        "### Loading Models"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-04-29T13:47:11.945605Z",
          "start_time": "2019-04-29T13:47:11.110004Z"
        },
        "id": "2xhvK0Zj98Zq"
      },
      "source": [
        "model = models.resnet50(pretrained=False).to(device)\n",
        "model.fc = nn.Sequential(\n",
        "    nn.Linear(2048, 128),\n",
        "    nn.ReLU(inplace=True),\n",
        "    nn.Linear(128, 2)).to(device)\n",
        "model.load_state_dict(torch.load('models/cva_weights.h5'))\n",
        "model.eval() # stop training model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J-W0-scw98Zr"
      },
      "source": [
        "### Make predictions on test images"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-04-29T13:47:15.486794Z",
          "start_time": "2019-04-29T13:47:15.441762Z"
        },
        "id": "Yg5NP4fr98Zr"
      },
      "source": [
        "validation_img_paths = [\"data/cva_data/validation/arepas/00000165.jpg\",\n",
        "                        \"data/cva_data/validation/cubanos/00000037.jpg\",\n",
        "                        \"data/cva_data/validation/cubanos/00000061.jpg\",\n",
        "                        \"data/cva_data/validation/arepas/00000003.jpeg\"]\n",
        "img_list = [Image.open(img_path) for img_path in validation_img_paths]\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-04-29T13:54:38.976494Z",
          "start_time": "2019-04-29T13:54:37.385606Z"
        },
        "id": "mGkOZaCd98Zs"
      },
      "source": [
        "validation_batch = torch.stack([data_transforms['validation'](img).to(device)\n",
        "                                for img in img_list])\n",
        " \n",
        "pred_logits_tensor = model(validation_batch)\n",
        "pred_probs = F.softmax(pred_logits_tensor, dim=1).cpu().data.numpy()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-04-29T13:54:40.651430Z",
          "start_time": "2019-04-29T13:54:40.088288Z"
        },
        "id": "nyEbGi4N98Zt"
      },
      "source": [
        "fig, axs = plt.subplots(1, len(img_list), figsize=(20, 5))\n",
        "for i, img in enumerate(img_list):\n",
        "    ax = axs[i]\n",
        "    ax.axis('off')\n",
        "    ax.set_title(\"{:.0f}% Arepa, {:.0f}% Cubano\".format(100*pred_probs[i,0],\n",
        "                                                          100*pred_probs[i,1]))\n",
        "    ax.imshow(img)\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F3f9DSrp98Zt"
      },
      "source": [
        "# CNN from Scratch"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IFfDZV5a98Zu"
      },
      "source": [
        "###  `Datasets` and `Dataloaders`\n",
        "\n",
        "In PyTorch, you'll usually create or import a `Dataset` subclass to represent your data. Once you've done that, you can use it to instantiate a `Dataloader` object which allows you to easily iterate over your training set in `BATCH_SIZE` chunks."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-05-01T15:23:01.483803Z",
          "start_time": "2019-05-01T15:22:52.888024Z"
        },
        "id": "rDY2WCgj98Zu"
      },
      "source": [
        "image_size = 28\n",
        "num_classes = 10\n",
        "num_channels = 1\n",
        "batch_size = 64\n",
        "id_to_label = {\n",
        "    0 :'T-shirt/top',\n",
        "    1 :'Trouser',\n",
        "    2 :'Pullover',\n",
        "    3 :'Dress',\n",
        "    4 :'Coat',\n",
        "    5 :'Sandal',\n",
        "    6 :'Shirt',\n",
        "    7 :'Sneaker',\n",
        "    8 :'Bag',\n",
        "    9 :'Ankle boot'}\n",
        "\n",
        "\n",
        "class FashionDataset(Dataset):\n",
        "\n",
        "    def __init__(self, path, \n",
        "                 image_size, num_channels, image_transform=None):\n",
        "        self.num_channels = num_channels\n",
        "        self.image_size = image_size\n",
        "        self.image_transform = image_transform\n",
        "        data_df = pd.read_csv(path)\n",
        "        self.X = data_df.values[:, 1:]\n",
        "        self.X = self.X.reshape(-1, image_size, image_size, num_channels)\n",
        "        self.X = self.X.astype('float32')\n",
        "        self.y = data_df.values[:, 0]\n",
        "        \n",
        "    def __getitem__(self, index):\n",
        "        batch_X, batch_y = self.X[index], self.y[index]\n",
        "        if self.image_transform is not None:\n",
        "            batch_X = self.image_transform(batch_X)\n",
        "        return batch_X, batch_y\n",
        "\n",
        "    def __len__(self):\n",
        "        return len(self.X)\n",
        "\n",
        "# This simple transform coverts the image from an numpy array\n",
        "# to a PyTorch tensor and remaps its values from 0-255 to 0-1. \n",
        "# Many other types of transformations are available, and they \n",
        "# can easily be composed into a pipeline. For more info see: \n",
        "# https://pytorch.org/docs/stable/torchvision/transforms.html\n",
        "image_transform = transforms.Compose([transforms.ToTensor()])\n",
        "\n",
        "train_dataset = FashionDataset(\n",
        "    'fashionmnist/fashion-mnist_train.csv', \n",
        "    image_size, \n",
        "    num_channels, \n",
        "    image_transform)\n",
        "\n",
        "train_dataloader = DataLoader(\n",
        "    dataset=train_dataset,\n",
        "    batch_size=batch_size,\n",
        "    shuffle=True)\n",
        "\n",
        "val_dataset = FashionDataset(\n",
        "    'fashionmnist/fashion-mnist_test.csv', \n",
        "    image_size, \n",
        "    num_channels, \n",
        "    image_transform)\n",
        "\n",
        "val_dataloader = DataLoader(\n",
        "    dataset=train_dataset,\n",
        "    batch_size=batch_size)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "87tWdYgJ98Zv"
      },
      "source": [
        "### Show some examples from the training set"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-05-02T01:58:53.841592Z",
          "start_time": "2019-05-02T01:58:53.065507Z"
        },
        "id": "G3_1m7hh98Zw"
      },
      "source": [
        "def plot_images(data, labels, image_size):\n",
        "    fig, axes = plt.subplots(\n",
        "        1, data.shape[0], figsize=(16, 4), \n",
        "        subplot_kw={'xticks':[], 'yticks':[]},\n",
        "        gridspec_kw=dict(hspace=0.1, wspace=0.1))\n",
        "    for i, ax in enumerate(axes.flat):\n",
        "        ax.imshow(data[i].reshape(image_size, image_size), cmap='binary')\n",
        "        ax.set_xlabel(labels[i])\n",
        "        \n",
        "images = train_dataset.X[:10]\n",
        "class_ids = train_dataset.y[:10]\n",
        "class_labels = [id_to_label[class_id] for class_id in class_ids]\n",
        "plot_images(images, class_labels, image_size)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wiTMWwoN98Zy"
      },
      "source": [
        "## Modeling"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lHn8Z9og98Zy"
      },
      "source": [
        "This baseline network uses a single convolution with only 4 filters. Because of the simplicity of our dataset, it still manages to achieve nearly 90% accuracy after only 5 epochs.\n",
        "\n",
        "Experiment and see if you can increase the accuracy on the validation set to above 95%.\n",
        "\n",
        "Things you might try:\n",
        "\n",
        "* Increasing the number of filters per convolution.\n",
        "* Adding more convolutions.\n",
        "* Adding a `BatchNorm2d` layer after `Conv2d`.\n",
        "* Increasing the number of epochs.\n",
        "* Changing the kernel size.\n",
        "* Using different types of pooling or using stride > 1 in convolutional layers instead of pooling.\n",
        "\n",
        "You might also find the [PyTorch API reference](https://pytorch.org/docs/stable/nn.html) useful."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mrKe0kZU98Zy"
      },
      "source": [
        "### Determining Output Size after Convolution and MaxPool\n",
        "\n",
        "When dealing with any neural net model, we have to ensure that each one of our layers is receiving the correct input size from its previous layers. This is easier said than done as, in Computer Vision, we have to convert from 2D layers such as convolutions, batch normalizations, and max pooling layers to communicate properly with a linear layer that follows. To do this, we have to ensure that our calculations for the **input** to each layer is correct when we instantiate our model.\n",
        "\n",
        "Linear layers are simple, the output size is stated when we instantiate. The difficulty lies in Conv2D layers since PyTorch calculates the output size using a formula and we need to determine what that size is by hand if we need it to pass it as input for other layers.\n",
        "\n",
        "To determine the output size after 2D layers we usually have to do it separately for our height and width, but since we are dealing with square images they will give us the same result. Furthermore, we'll remove any values that are set to zero in our model to reduce the complexity of the equation. For more information on output size for different functions, refer to the PyTorch API.\n",
        "\n",
        "#### Conv2D\n",
        "\n",
        "$$H_{out} = \\frac{H_{in} - kernel + 2 * padding}{stride} + 1$$\n",
        "\n",
        "\n",
        "#### MaxPool2D\n",
        "$$H_{out} = \\frac{H_{in} - kernel + 2*padding}{stride} + 1$$\n",
        "\n",
        "\n",
        "\n",
        "Let's use the numbers from our `FashionModel` (see below) to determine what the outputs are after our first combo of `Conv2d` and `MaxPool2d`\n",
        "\n",
        "Since our images are square, (28x28) we only need to do this calcluation once.\n",
        "\n",
        "**Size of image:** 28\n",
        "\n",
        "**Output size after Conv2d:**\n",
        "$$H_{out} = \\frac{28 - 3 + 2*1}{1} + 1 = 28$$\n",
        "\n",
        "**Output size after MaxPool2d:**\n",
        "$$H_{out} = \\frac{28 - 2 + 2*0}{2} + 1 = 14$$\n",
        "\n",
        "\n",
        "If passing to a linear layer we need to multiply `output_channels` from our `Conv2d` times the `output_size_height` and `output_size_width` (in our case they are the same) to properly rescale our tensor. \n",
        "\n",
        "**Output channels (after Conv2d):** 16\n",
        "\n",
        "**MaxPool Output Size:** 14 (same for both height and width)\n",
        "\n",
        "Thus, our linear layer needs an input of size `14 * 14 * 16`\n",
        "\n",
        "To generalize, the  need `output_size_width * output_size_height * output_channels` \n",
        "\n",
        "For more info see [here.](https://www.quora.com/How-can-I-calculate-the-size-of-output-of-convolutional-layer)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-05-02T02:14:36.037653Z",
          "start_time": "2019-05-02T02:14:36.019257Z"
        },
        "id": "ksWzfB3m98Zy"
      },
      "source": [
        "class FashionModel(nn.Module):\n",
        "    \n",
        "    def __init__(self, num_channels, num_classes):\n",
        "        super().__init__()\n",
        "        self.conv_1 = nn.Sequential(\n",
        "            nn.Conv2d(1, 16, kernel_size=3, stride=1, padding=1),\n",
        "            nn.BatchNorm2d(16),\n",
        "            nn.ReLU(inplace=True),\n",
        "            nn.MaxPool2d(kernel_size=2, stride=2, padding=0))\n",
        "        # Add more conv2d layers\n",
        "        # recall that the input channel size for the new layer is 16! \n",
        "        # Make the outputs 32 and 64 to create 3 total convolution layers\n",
        "        \n",
        "        \n",
        "        # ------------- CODE GOES HERE ------------- #\n",
        "        self.conv_2 = nn.Sequential(\n",
        "          nn.Conv2d(16, 32, kernel_size=3, stride=1, padding=1),\n",
        "          nn.BatchNorm2d(32),\n",
        "          nn.ReLU(inplace=True),\n",
        "          nn.MaxPool2d(kernel_size=2, stride=2, padding=0))\n",
        "                \n",
        "        # to determine the correct inputs for the last linear layer you'll need the formulas: \n",
        "        # output_size of conv2d = (input_width - kernel_size + 2*padding)/stride + 1\n",
        "        # output_size of maxpool = (output_size of conv2d - kernel_size + 2*padding)/stride + 1\n",
        "        \n",
        "        # See cell above for more detail\n",
        "        \n",
        "        # If adding an extra conv layers, the input into nn.Linear needs to change!\n",
        "        self.linear = nn.Linear(7 * 7 * 32, num_classes)\n",
        "        \n",
        "    def forward(self, x):\n",
        "        x = self.conv_1(x)\n",
        "        # Uncomment as more convolution layers are added\n",
        "        x = self.conv_2(x)\n",
        "        # x = self.conv_3(x)\n",
        "        x = x.reshape(x.size(0), -1)\n",
        "        x = self.linear(x)\n",
        "        return x\n",
        "\n",
        "\n",
        "# Instantiate the model.    \n",
        "model = FashionModel(num_channels, num_classes)\n",
        "\n",
        "# Send the model's tensors to the GPU (if available).\n",
        "model = model.to(device)    "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xMphQPq_98Zz"
      },
      "source": [
        "## Training"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-05-02T02:14:51.121614Z",
          "start_time": "2019-05-02T02:14:42.815674Z"
        },
        "id": "v63uptok98Z0"
      },
      "source": [
        "num_epochs = 10\n",
        "log_freq = 100\n",
        "checkpoint_path = 'checkpoint.pickle'\n",
        "\n",
        "\n",
        "optimizer = torch.optim.Adam(model.parameters())\n",
        "criterion = nn.CrossEntropyLoss()\n",
        "for epoch in range(1, num_epochs + 1):\n",
        "    model.train()  # Switch to training mode.\n",
        "    print(f'Starting epoch {epoch}.')\n",
        "    epoch_start_time = time.time()\n",
        "    running_loss = 0.0\n",
        "    running_accuracy = 0.0\n",
        "    for batch_id, (batch_X, batch_y) in enumerate(train_dataloader):\n",
        "        batch_X = batch_X.to(device)\n",
        "        batch_y = batch_y.to(device)\n",
        "        output = model(batch_X)\n",
        "        loss = criterion(output, batch_y)\n",
        "        optimizer.zero_grad()\n",
        "        loss.backward()\n",
        "        optimizer.step()\n",
        "        \n",
        "        # Periodically print the loss and prediction accuracy.\n",
        "        running_loss += loss.item()\n",
        "        y_pred = output.argmax(dim=-1)\n",
        "        running_accuracy += accuracy_score(batch_y.cpu(), y_pred.cpu())\n",
        "        if batch_id % log_freq == log_freq - 1:\n",
        "            average_loss = running_loss / log_freq\n",
        "            average_accuracy = running_accuracy / log_freq\n",
        "            print(f'Mini-batch: {batch_id + 1}/{len(train_dataloader)} '\n",
        "                  f'Loss: {average_loss:.5f} Accuracy: {average_accuracy:.5f}')\n",
        "            running_loss = 0.0\n",
        "            running_accuracy = 0.0\n",
        "            \n",
        "    # Log elapsed_time for the epoch.            \n",
        "    elapsed_time = time.time() - epoch_start_time\n",
        "    print(f'\\nEpoch {epoch} completed in {elapsed_time // 60:.0f} minutes '\n",
        "          f'{elapsed_time % 60:.0f} seconds.')\n",
        "    \n",
        "    # Calculate and log loss on validation set.\n",
        "    with torch.no_grad():\n",
        "        model.eval()\n",
        "        running_loss = 0.0\n",
        "        running_accuracy = 0.0\n",
        "        for batch_id, (batch_X, batch_y) in enumerate(val_dataloader):\n",
        "            batch_X = batch_X.to(device)\n",
        "            batch_y = batch_y.to(device)\n",
        "            output = model(batch_X)\n",
        "            loss = criterion(output, batch_y)\n",
        "            running_loss += loss.item()\n",
        "            y_pred = output.argmax(dim=-1)  \n",
        "            running_accuracy += accuracy_score(batch_y.cpu(), y_pred.cpu())\n",
        "        average_loss = running_loss / len(val_dataloader)\n",
        "        average_accuracy = running_accuracy / len(val_dataloader)\n",
        "        print(f'Val Loss: {average_loss:.5f} Val Accuracy: {average_accuracy:.5f}\\n')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eAuXhyQb98Z6"
      },
      "source": [
        "# Semi-Supervised Learning - Style Transfer\n",
        "\n",
        "Style transfer, to me, is one of the coolest discoveries in the computer vision and deep learning community from the past few years. In essence, it allows us to take the \"content\" from an image (shapes, objects, arrangements) and reproduce a new target that is in the \"style\" (style, colors, textures) of another. \n",
        "\n",
        "We'll be taking inspiration from the paper, [Image Style Transfer Using Convolutional Neural Networks, by Gatys](https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Gatys_Image_Style_Transfer_CVPR_2016_paper.pdf), and implementing the model in PyTorch.\n",
        "\n",
        "In the paper, style transfer uses the features found in the 19-layer VGG Network, which are comprised of a series of convolutional and pooling layers, and a few fully-connected layers. (Recall that this is similar to many of the computer vision models discussed earlier) \n",
        "\n",
        "In the image below, the convolutional layers are named by stack and their order in the stack.\n",
        "\n",
        "For example, `Conv_1_1` is the first convolutional layer that an image is passed through in the *first* stack. `Conv_2_1` is the first convolutional layer in the *second* stack. The deepest convolutional layer in the network is `Conv_5_4`.\n",
        "\n",
        "<img src='https://github.com/robert-alvarez/pytorch_tutorial/blob/master/data/img/vgg19_convlayers.png?raw=1' width=80% />\n",
        "\n",
        "Style transfer relies on separating the content and style of an image. To do so, we aim to create a new **target** image which should contain our desired content and style components.\n",
        "\n",
        "**Note:**\n",
        "* objects and their arrangement are similar to that of the **content image**\n",
        "* style, colors, and textures are similar to that of the **style image**\n",
        "\n",
        "<img src='https://github.com/robert-alvarez/pytorch_tutorial/blob/master/data/img/style-transfer-example.jpg?raw=1' width=80% />"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kH93VNUL98Z6"
      },
      "source": [
        "## Load VGG19\n",
        "\n",
        "VGG19 is split into two portions:\n",
        "* `vgg19.features` - contains all the convolutional and pooling layers\n",
        "* `vgg19.classifier` - contains the three fully connected layers layers at the end\n",
        "\n",
        "We only need the `features` portion, which we're going to load in and \"freeze\" the weights of. This is similar to what we did for our transfer learning section."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hQXtdum498Z7"
      },
      "source": [
        "# get the \"features\" portion of VGG19\n",
        "vgg = models.vgg19(pretrained=True).features\n",
        "\n",
        "# freeze all VGG parameters since we're only optimizing the target image\n",
        "for param in vgg.parameters():\n",
        "    param.requires_grad_(False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3KCR6u8L98Z9"
      },
      "source": [
        "# move the model to GPU (if available)\n",
        "vgg.to(device)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rVGVbkt_98Z-"
      },
      "source": [
        "### Load in Content and Style Images\n",
        "\n",
        "Load in any images you want! The code below is a helper function for loading in any type and size of image. The `load_image` function also converts images to normalized Tensors."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "emPYYCP398Z_"
      },
      "source": [
        "def load_image(img_path, max_size=400, shape=None):\n",
        "    ''' Load in and transform an image, making sure the image\n",
        "       is <= 400 pixels in the x-y dims.'''\n",
        "    if \"http\" in img_path:\n",
        "        response = requests.get(img_path)\n",
        "        image = Image.open(BytesIO(response.content)).convert('RGB')\n",
        "    else:\n",
        "        image = Image.open(img_path).convert('RGB')\n",
        "    \n",
        "    # large images will slow down processing\n",
        "    if max(image.size) > max_size:\n",
        "        size = max_size\n",
        "    else:\n",
        "        size = max(image.size)\n",
        "    \n",
        "    if shape is not None:\n",
        "        size = shape\n",
        "        \n",
        "    in_transform = transforms.Compose([\n",
        "                        transforms.Resize(size),\n",
        "                        transforms.ToTensor(),\n",
        "                        transforms.Normalize((0.485, 0.456, 0.406), \n",
        "                                             (0.229, 0.224, 0.225))])\n",
        "\n",
        "    # discard the transparent, alpha channel (that's the :3) and add the batch dimension\n",
        "    image = in_transform(image)[:3,:,:].unsqueeze(0)\n",
        "    \n",
        "    return image"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4z1U5NhX98Z_"
      },
      "source": [
        "# load in content and style image\n",
        "content = load_image('/content/data/img/Nala.jpg').to(device)\n",
        "# Resize style to match content, makes code easier\n",
        "style = load_image('/content/data/img/style_images/starry_night_full.jpg', shape=content.shape[-2:]).to(device)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QIzfP7-M98aA"
      },
      "source": [
        "# helper function for un-normalizing an image \n",
        "# and converting it from a Tensor image to a NumPy image for display\n",
        "def im_convert(tensor):\n",
        "    \"\"\" Display a tensor as an image. \"\"\"\n",
        "    \n",
        "    image = tensor.to(\"cpu\").clone().detach()\n",
        "    image = image.numpy().squeeze()\n",
        "    image = image.transpose(1,2,0)\n",
        "    image = image * np.array((0.229, 0.224, 0.225)) + np.array((0.485, 0.456, 0.406))\n",
        "    image = image.clip(0, 1)\n",
        "\n",
        "    return image"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8YQn4XdO98aB"
      },
      "source": [
        "# display the images\n",
        "fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(20, 10))\n",
        "# content and style ims side-by-side\n",
        "ax1.imshow(im_convert(content))\n",
        "ax2.imshow(im_convert(style))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M_SLPHnw98aC"
      },
      "source": [
        "## VGG19 Layers\n",
        "\n",
        "To get the content and style representations of an image, we have to pass an image forward throug the VGG19 network until we get to the desired layer(s) and then get the output from that layer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qjOozSKG98aC"
      },
      "source": [
        "# print out VGG19 model so you can see the names of all the layers\n",
        "print(vgg)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EOKXQcXC98aE"
      },
      "source": [
        "## Content and Style Features\n",
        "\n",
        "Below, complete the mapping of layer names to the names found in the paper for the _content representation_ and the _style representation_."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-P8dLNmi98aE"
      },
      "source": [
        "def get_features(image, model, layers=None):\n",
        "    \"\"\" Run an image forward through a model and get the features for \n",
        "        a set of layers. Default layers are for VGGNet matching Gatys et al (2016)\n",
        "    \"\"\"\n",
        "    \n",
        "    ## Here we gather the layers need to preserve style and content of an image\n",
        "    if layers is None:\n",
        "        layers = {'1': 'relu1_1',\n",
        "                  '6': 'relu2_1', \n",
        "                  '11': 'relu3_1', \n",
        "                  '20': 'relu4_1',\n",
        "                  '13': 'relu3_2',  ## content representation\n",
        "                  '29': 'relu5_1'}\n",
        "        \n",
        "    features = {}\n",
        "    x = image\n",
        "    # model._modules is a dictionary holding each module in the model\n",
        "    for name, layer in model._modules.items():\n",
        "        x = layer(x)\n",
        "        if name in layers:\n",
        "            features[layers[name]] = x\n",
        "            \n",
        "    return features"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U_9ncp6S98aH"
      },
      "source": [
        "## Gram Matrix \n",
        "\n",
        "The output of every convolutional layer is a Tensor with dimensions associated with the `batch_size`, a depth, `d` and some height and width (`h`, `w`). The Gram matrix of a convolutional layer can be calculated as follows:\n",
        "* Get the depth, height, and width of a tensor using `batch_size, d, h, w = tensor.size`\n",
        "* Reshape that tensor so that the spatial dimensions are flattened\n",
        "* Calculate the gram matrix by multiplying the reshaped tensor by it's transpose "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3l1_Lb-X98aH"
      },
      "source": [
        "def gram_matrix(tensor):\n",
        "    \"\"\" Calculate the Gram Matrix of a given tensor \n",
        "        Gram Matrix: https://en.wikipedia.org/wiki/Gramian_matrix\n",
        "    \"\"\"\n",
        "    \n",
        "    # get the batch_size, depth, height, and width of the Tensor\n",
        "    _, d, h, w = tensor.size()\n",
        "    \n",
        "    # reshape so we're multiplying the features for each channel\n",
        "    tensor = tensor.view(d, h * w)\n",
        "    \n",
        "    # calculate the gram matrix\n",
        "    gram = tensor @ tensor.t()\n",
        "    \n",
        "    return gram "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tJIQ9riA98aK"
      },
      "source": [
        "## We're Almost there!\n",
        "\n",
        "Here's what our helper functions do:\n",
        "\n",
        "* Extract the features for our content and style images from VGG19\n",
        "* Compute the Gram Matrix for a given convolutional layer\n",
        "\n",
        "\n",
        "What we need:\n",
        "* Put it all together!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1b8fo8gg98aK"
      },
      "source": [
        "# get content and style features only once before training\n",
        "content_features = get_features(content, vgg)\n",
        "style_features = get_features(style, vgg)\n",
        "\n",
        "# calculate the gram matrices for each layer of our style representation\n",
        "style_grams = {layer: gram_matrix(style_features[layer]) for layer in style_features}\n",
        "\n",
        "# create a third \"target\" image and prep it for change\n",
        "# it is a good idea to start off with the target as a copy of our *content* image\n",
        "# then iteratively change its style\n",
        "target = content.clone().requires_grad_(True).to(device)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kzb2iaqA98aL"
      },
      "source": [
        "## Loss and Weights\n",
        "\n",
        "#### Individual Layer Style Weights\n",
        "\n",
        "In the script below, we have the option to weight the style representation at each relevant layer. This will allow us to fine tune what effect size we want for each layer - earlier layers have larger style artifacts and later layers place emphasis on smaller features. Remember, each layer is a different size and by combining them we can create multi-scale style representations.\n",
        "\n",
        "The paper suggests using a range between 0-1 to weight the layers.\n",
        "\n",
        "#### Content and Style Weight\n",
        "\n",
        "The paper defines a **style ratio** of $\\alpha/\\beta$, where  $\\alpha$ is the `content_weight` and  $\\beta$ is the `style_weight`. This ratio will affect how _stylized_ the final image is. It's recommended that to leave the content_weight = 1 and set the style_weight to achieve the ratio needed for a desired effect style. Note that this is not exact science, there will be lots of tuning of the ratio and the weights of the layers to get a result we're pleased with.\n",
        "\n",
        "Remember - the reason this is called \"Semi-Supervised\" is because there is no right answer. We decide when to stop the training based on intermediate results we plot and stop when we are happy with what we see."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-GXW_90X98aL"
      },
      "source": [
        "# weights for each style layer \n",
        "# weighting earlier layers more will result in *larger* style artifacts\n",
        "# notice we are excluding `conv4_2` our content representation\n",
        "style_weights = {'relu1_1': 0.75,\n",
        "                 'relu2_1': 1.0,\n",
        "                 'relu3_1': 0.4,\n",
        "                 'relu4_1': 0.3,\n",
        "                 'relu5_1': 0.2}\n",
        "\n",
        "content_weight = 1  # alpha\n",
        "style_weight = 1e7  # beta"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U-Aa1VAY98aM"
      },
      "source": [
        "## Update the Target & Calculate Losses\n",
        "\n",
        "Like in every training loop we've seen before, we need to decide on how many passes we want to do on our model (via gradient descent). The difference here is that we will be changing our **target** image and nothing about the VGG19 model or our original content and style images. Since this is semi-supervised, the number of steps to choose is up to you. Keep in mind that after ~50,000 steps you probably won't see any noticeable differences in the images and by using ~2,000 steps you can see early on whether the style ratio ($\\alpha/\\beta$) is giving the desired effect. \n",
        "\n",
        "Experiment with different weights or images to see some really cool effects!\n",
        "\n",
        "#### Content Loss\n",
        "\n",
        "The content loss will be the mean squared difference between the target and content features at layer `conv4_2`. This can be calculated as follows: (see paper)\n",
        "```\n",
        "content_loss = torch.mean(torch.abs(target_features['conv4_2'] - content_features['conv4_2']))\n",
        "```\n",
        "\n",
        "#### Style Loss\n",
        "\n",
        "The style loss is calculated in a similar way, but we have to iterate through the layers specified by name in our dictionary `style_weights`. \n",
        "\n",
        "> Calculate the gram matrix for the target image, `target_gram` and style image `style_gram` at each of these layers and compare those gram matrices, calculating the `layer_style_loss`. \n",
        "\n",
        "#### Total Loss\n",
        "\n",
        "Finally, the total loss is calculated by adding up the individual style and content losses and weighting them with the specified alpha and beta values chosen.\n",
        "\n",
        "Intermittently, we'll print out an intermediate image and its loss - don't be alarmed if the loss is very large! It takes some time for an image's style to change and you should focus on the appearance of your target image rather than any loss value, but we should still be seeing the loss go down over time."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tydP-K9c98aM"
      },
      "source": [
        "# for displaying the target image, intermittently\n",
        "show_every = 500\n",
        "\n",
        "# iteration hyperparameters\n",
        "optimizer = torch.optim.Adam([target], lr=0.003)\n",
        "steps = 20_000  # decide how many iterations to update your image\n",
        "\n",
        "for step in range(1, steps+1):\n",
        "    \n",
        "    # get the features from your target image\n",
        "    target_features = get_features(target, vgg)\n",
        "    \n",
        "    # the content loss\n",
        "    content_loss = torch.mean(torch.abs(target_features['relu3_2'] - content_features['relu3_2']))\n",
        "    \n",
        "    # the style loss\n",
        "    # initialize the style loss to 0\n",
        "    style_loss = 0\n",
        "    # then add to it for each layer's gram matrix loss\n",
        "    for layer in style_weights:\n",
        "        # get the \"target\" style representation for the layer\n",
        "        target_feature = target_features[layer]\n",
        "        target_gram = gram_matrix(target_feature)\n",
        "        _, d, h, w = target_feature.shape\n",
        "        # get the \"style\" style representation\n",
        "        style_gram = style_grams[layer]\n",
        "        # the style loss for one layer, weighted appropriately\n",
        "        layer_style_loss = style_weights[layer] * torch.mean((target_gram - style_gram)**2)\n",
        "        # add to the style loss\n",
        "        style_loss += layer_style_loss / (d * h * w)\n",
        "        \n",
        "    # calculate the *total* loss\n",
        "    total_loss = content_weight * content_loss + style_weight * style_loss\n",
        "    \n",
        "    # update your target image\n",
        "    optimizer.zero_grad()\n",
        "    total_loss.backward()\n",
        "    optimizer.step()\n",
        "    \n",
        "    # display intermediate images and print the loss\n",
        "    if  step % show_every == 0:\n",
        "        print('Total loss: ', total_loss.item())\n",
        "        plt.imshow(im_convert(target))\n",
        "        plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "US9otQ0e98aN"
      },
      "source": [
        "## Display the Target Image"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dVB8O5_998aO"
      },
      "source": [
        "# display content and target image\n",
        "fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(20, 10))\n",
        "ax1.imshow(im_convert(content))\n",
        "ax2.imshow(im_convert(target))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "r9IvOCk598aP"
      },
      "source": [
        "# display style and target image\n",
        "fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(20, 10))\n",
        "ax1.imshow(im_convert(style))\n",
        "ax2.imshow(im_convert(target))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Vo6F3GOJrISA"
      },
      "source": [
        "plt.imshow(im_convert(target))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XdzRCNyauoww"
      },
      "source": [
        "fig, ax = plt.subplots(1,1,figsize=(20,20))\n",
        "ax.imshow(im_convert(target))\n",
        "ax.spines['top'].set_visible(False)\n",
        "ax.spines['left'].set_visible(False)\n",
        "ax.spines['bottom'].set_visible(False)\n",
        "ax.spines['right'].set_visible(False)\n",
        "ax.set_xticks([])\n",
        "ax.set_yticks([])\n",
        "plt.savefig(\"nala_mod.png\",bbox_inches=\"tight\",pad_inches=0.02,dpi=300)\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "a_ZqoJnWyDEz"
      },
      "source": [
        "# display fully stylized image vs style\n",
        "file_path = '/content/data/img/nala_mod.png'\n",
        "fig, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(20, 10))\n",
        "ax1.imshow(im_convert(content))\n",
        "ax2.imshow(im_convert(style))\n",
        "ax3.imshow(Image.open(file_path))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_UZVR5_V98aR"
      },
      "source": [
        "---\n",
        "# RNN Character Level Generation\n",
        "\n",
        "Text generation is a fun way to familiarize yourself with Recurrent Neural Nets.\n",
        "\n",
        "In this notebook, we will deal with **character-level** text generation and why they can be just as useful as word-level text generation.\n",
        "\n",
        "For this example, we'll be using text files to generate code similar to our input. In other words, if we put in Trump tweets, our generator should output words that sound like Trump.\n",
        "\n",
        "### Our current understanding of RNNs\n",
        "![](https://github.com/robert-alvarez/pytorch_tutorial/blob/master/data/img/LSTM_next_character.png?raw=1)\n",
        "\n",
        "\n",
        "### Reminder\n",
        "\n",
        "![](https://github.com/robert-alvarez/pytorch_tutorial/blob/master/data/img/rnn_unrolling.png?raw=1)\n",
        "\n",
        "#### \"RNNs have a hidden state that feeds back into the cell at the next time step\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2_GV03CE98aR"
      },
      "source": [
        "## What is actually going on?\n",
        "\n",
        "### Example with sequence length 5"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2W1SYysW98aR"
      },
      "source": [
        "![](https://github.com/robert-alvarez/pytorch_tutorial/blob/master/data/img/rnn_forward.png?raw=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-s_MsXJM98aR"
      },
      "source": [
        "### What about the backwards pass?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V3fvQzdk98aS"
      },
      "source": [
        "![](https://github.com/robert-alvarez/pytorch_tutorial/blob/master/data/img/rnn_backward.png?raw=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_AXEdHUq98aS"
      },
      "source": [
        "## Preprocessing"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9licgHnh98aS"
      },
      "source": [
        "# install unidecode\n",
        "!pip install unidecode"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-04-30T21:57:47.121064Z",
          "start_time": "2019-04-30T21:57:47.113879Z"
        },
        "id": "EwM19pvC98aU"
      },
      "source": [
        "# Read in text and change unicode characters to ASCII\n",
        "import unidecode\n",
        "import string\n",
        "import random\n",
        "import re\n",
        "\n",
        "all_characters = string.printable\n",
        "n_characters = len(all_characters)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-04-30T21:57:48.885409Z",
          "start_time": "2019-04-30T21:57:48.873437Z"
        },
        "id": "5nkrsGUh98aV"
      },
      "source": [
        "# read in file to train RNN\n",
        "file = unidecode.unidecode(open('data/shakespeare.txt').read())\n",
        "file_len = len(file)\n",
        "print(f'file_len = {file_len}')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ReHlckvd98aW"
      },
      "source": [
        "To give our model inputs from this large string of text, we'll split it up into chunks"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-04-30T21:57:52.871182Z",
          "start_time": "2019-04-30T21:57:52.861252Z"
        },
        "id": "lKIIRWCE98aW"
      },
      "source": [
        "chunk_len = 400\n",
        "\n",
        "def random_chunk():\n",
        "    start_index = random.randint(0, file_len - chunk_len)\n",
        "    end_index = start_index + chunk_len + 1\n",
        "    return file[start_index:end_index]\n",
        "\n",
        "print(random_chunk())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pi0rBFSe98aW"
      },
      "source": [
        "## Build Model\n",
        "\n",
        "This model will take as input the character for step $t$ and is expected to output the next character for step $t+1$. There are three layers - one linear layer that encodes the input character into an internal state, one GRU layer (which may itself have multiple layers) that operates on that internal state and a hidden state, and a decoder layer that outputs the probability distribution."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-04-30T21:58:21.474477Z",
          "start_time": "2019-04-30T21:58:21.464818Z"
        },
        "id": "GDk8bHgi98aX"
      },
      "source": [
        "class RNN(nn.Module):\n",
        "    def __init__(self, input_size, hidden_size, output_size, n_layers=1):\n",
        "        super(RNN, self).__init__()\n",
        "        self.input_size = input_size\n",
        "        self.hidden_size = hidden_size\n",
        "        self.output_size = output_size\n",
        "        self.n_layers = n_layers\n",
        "        \n",
        "        self.encoder = nn.Embedding(input_size, hidden_size)\n",
        "        self.gru = nn.GRU(hidden_size, hidden_size, n_layers)\n",
        "        self.decoder = nn.Linear(hidden_size, output_size)\n",
        "        \n",
        "    def forward(self, input, hidden):\n",
        "        output = self.encoder(input.view(1, -1))\n",
        "        output, hidden = self.gru(output.view(1, 1, -1), hidden)\n",
        "        output = self.decoder(output.view(1, -1))\n",
        "        \n",
        "        return output, hidden\n",
        "    \n",
        "    def init_hidden(self):\n",
        "        return Variable(torch.randn(self.n_layers, 1, self.hidden_size))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fXw2G1Fu98aZ"
      },
      "source": [
        "## Inputs and Targets\n",
        "\n",
        "Now that we've defined our model, we need to give it both input data, via our chunks, and our target data. Each character is one-hot encoded to the vocab size "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-04-30T21:58:24.169426Z",
          "start_time": "2019-04-30T21:58:24.161142Z"
        },
        "id": "JYBv5jbj98aZ"
      },
      "source": [
        "def char2tensor(string):\n",
        "    tensor = torch.zeros(len(string)).long()\n",
        "    for char in range(len(string)):\n",
        "        if string[char] in all_characters:\n",
        "            tensor[char] = all_characters.index(string[char])\n",
        "        else:\n",
        "            tensor[char] = 94 #predict space if character unknown\n",
        "        \n",
        "    return Variable(tensor)\n",
        "\n",
        "# Let's see it in action.\n",
        "print(char2tensor('Podium0123abczABC'))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KQeU8ig398aa"
      },
      "source": [
        "Now that we can generate chunks of data, we can build our inputs and targets.\n",
        "\n",
        "Our inputs will be all of the chunk except for the last letter. \n",
        "\n",
        "Our target will be all of the chunk except for the first letter."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-04-30T21:58:26.734169Z",
          "start_time": "2019-04-30T21:58:26.721769Z"
        },
        "id": "-3WJANx798aa"
      },
      "source": [
        "def random_training_set():\n",
        "    chunk = random_chunk()\n",
        "    inp = char2tensor(chunk[:-1])\n",
        "    target = char2tensor(chunk[1:])\n",
        "    return inp, target"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cgVLzwWA98aa"
      },
      "source": [
        "## Evaluating the Model\n",
        "\n",
        "To evaluate the network we will feed one character at a time, use the outputs of the network as a probability distribution for the next character, and repeat. To start generation we pass a priming string to start building up the hidden state, from which we then generate one character at a time."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-04-30T21:58:29.035941Z",
          "start_time": "2019-04-30T21:58:28.959259Z"
        },
        "id": "rbdP6olM98ab"
      },
      "source": [
        "def evaluate(model, prime_str='A', predict_len=100, temperature=0.8):\n",
        "    \n",
        "    hidden = model.init_hidden()\n",
        "    prime_input = char2tensor(prime_str)\n",
        "    predicted = prime_str\n",
        "    \n",
        "    # use priming string to build up hidden state\n",
        "    \n",
        "    for p in range(len(prime_str) - 1):\n",
        "        _, hidden = model(prime_input[p], hidden)\n",
        "    inp = prime_input[-1]\n",
        "    \n",
        "    for p in range(predict_len):\n",
        "        output, hidden = model(inp, hidden)\n",
        "        \n",
        "        # Sample from the network as a multinomial distribution\n",
        "        output_dist = output.data.view(-1).div(temperature).exp()\n",
        "        top_i = torch.multinomial(output_dist, 1)[0]\n",
        "        \n",
        "        # Add predicted character to string and use as next input\n",
        "        predicted_char = all_characters[top_i]\n",
        "        predicted += predicted_char\n",
        "        inp = char2tensor(predicted_char)\n",
        "        \n",
        "    return predicted"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BCXSgyAq98ac"
      },
      "source": [
        "## Training"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-04-30T21:58:32.785485Z",
          "start_time": "2019-04-30T21:58:32.776492Z"
        },
        "id": "RFvGhOSG98ac"
      },
      "source": [
        "# helper function\n",
        "import time, math\n",
        "\n",
        "def time_since(since):\n",
        "    s = time.time() - since\n",
        "    m = math.floor(s/60)\n",
        "    s -= m*60\n",
        "    return '%dm %ds' % (m, s)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-04-30T21:58:35.268443Z",
          "start_time": "2019-04-30T21:58:35.251463Z"
        },
        "id": "S4VVSZk598ac"
      },
      "source": [
        "# The actual training part\n",
        "def train(inp, target):\n",
        "    hidden = model.init_hidden()\n",
        "    model.zero_grad()\n",
        "    loss = 0\n",
        "\n",
        "    for char in range(chunk_len):\n",
        "        output, hidden = model(inp[char], hidden)\n",
        "        loss += criterion(output, target[char].unsqueeze(0))\n",
        "\n",
        "    loss.backward()\n",
        "    model_optimizer.step()\n",
        "\n",
        "    return loss.data.item() / chunk_len"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MZGTd4Ta98ad"
      },
      "source": [
        "# parameters\n",
        "n_epochs = 1000\n",
        "print_every = 100\n",
        "plot_every = 10\n",
        "hidden_size = 256\n",
        "n_layers = 2\n",
        "learning_rate = 0.001\n",
        "\n",
        "# model declaration\n",
        "model = RNN(n_characters, hidden_size, n_characters, n_layers)\n",
        "model_optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)\n",
        "criterion = nn.CrossEntropyLoss()\n",
        "\n",
        "start = time.time()\n",
        "all_losses = []\n",
        "loss_avg = 0\n",
        "\n",
        "for epoch in range(1, n_epochs + 1):\n",
        "    loss = train(*random_training_set())       \n",
        "    loss_avg += loss\n",
        "\n",
        "    if epoch % print_every == 0:\n",
        "        print('[%s (%d %d%%) %.4f]' % (time_since(start), epoch, epoch / n_epochs * 100, loss))\n",
        "        print(evaluate(model, 'A ', 100), '\\n')\n",
        "\n",
        "    if epoch % plot_every == 0:\n",
        "        all_losses.append(loss_avg / plot_every)\n",
        "        loss_avg = 0"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eH0W9ZUD98ae"
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "import matplotlib.ticker as ticker\n",
        "%matplotlib inline\n",
        "\n",
        "plt.figure()\n",
        "plt.plot(all_losses)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RUFswDvk98af"
      },
      "source": [
        "### Load pre-trained models"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hmgQhWKUVJ8o"
      },
      "source": [
        "# model declaration\n",
        "hidden_size = 256\n",
        "billy = RNN(n_characters, hidden_size, n_characters, n_layers=2)\n",
        "billy.load_state_dict(torch.load('data/models/shakespeare_weights.h5'))\n",
        "cells = RNN(n_characters, hidden_size, n_characters, n_layers=2)\n",
        "cells.load_state_dict(torch.load('data/models/cell_weights.h5'))\n",
        "\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-05-01T12:59:59.874947Z",
          "start_time": "2019-05-01T12:59:59.537376Z"
        },
        "id": "jGtRIvOi98ah"
      },
      "source": [
        "# Evaluate Billy Shakespeare\n",
        "print(evaluate(billy, 'To be or not to be: ', predict_len=200, temperature=0.7))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2019-05-01T18:12:56.428857Z",
          "start_time": "2019-05-01T18:12:56.112249Z"
        },
        "id": "ZKYO0B8A98ai"
      },
      "source": [
        "# Evaluate NLP model of math biology latex file\n",
        "print(evaluate(cells, 'The equation ', predict_len=200, temperature=0.4))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nzJ43EHb98aj"
      },
      "source": [
        "# Resources\n",
        "\n",
        "[tensor images](https://hackernoon.com/learning-ai-if-you-suck-at-math-p4-tensors-illustrated-with-cats-27f0002c9b32)\n",
        "\n",
        "[alien vs predator](https://deepsense.ai/keras-vs-pytorch-avp-transfer-learning/)\n",
        "\n",
        "[unreasonable reffectiveness of neural nets](http://karpathy.github.io/2015/05/21/rnn-effectiveness/)\n",
        "\n",
        "[pytorch.org](https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html)\n",
        "\n",
        "[Richard Galvez](https://richardagalvez.github.io/)\n",
        "\n",
        "[Visualizing and Understanding Convolutional Networks](https://arxiv.org/pdf/1311.2901.pdf)\n",
        "\n",
        "[Image Style Transfer](https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Gatys_Image_Style_Transfer_CVPR_2016_paper.pdf)\n",
        "\n",
        "[Transfer Learning Image](https://www.researchgate.net/figure/Illustration-of-transfer-learning-concept-where-the-rst-layers-in-network-A-and-network-B_fig2_316748306)\n",
        "\n",
        "[Udacity](https://github.com/udacity/deep-learning-v2-pytorch)\n"
      ]
    }
  ]
}