notebooks/cis6930_week1_deep_learning_basics.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "cis6930-week1_deep-learning-basics.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pckknBHRY4cb"
      },
      "source": [
        "# CIS6930 Week 1: Deep Learning Basics (For students)\n",
        "\n",
        "---\n",
        "\n",
        "Preparation: Go to `Runtime > Change runtime type` and choose `GPU` for the hardware accelerator.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ukPqhjL8jGnm"
      },
      "source": [
        "## PyTorch Basics\n",
        "\n",
        "`torch.Tensor` is similar to `numpy.ndarray` in the sense that it \"wraps\" vector/matrix values with data type information. PyTorch implements similar operations as NumPy. \n",
        "\n",
        "Let's take a look at how we create `torch.Tensor` and convert `torch.Tensor` into `numpy.ndarray` or `list` in addition to moving the tensor data between CPUs and GPUs. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zvY-NLq2Yy47"
      },
      "source": [
        "import numpy as np\n",
        "import torch\n",
        "torch.__version__"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dUSLkZHyZKBa"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OHCafy6bY4H_"
      },
      "source": [
        "np.array([1, 2, 3])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7jPJ_tGGZC0J"
      },
      "source": [
        "torch.Tensor([1, 2, 3])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_c4cqhxcZE0e"
      },
      "source": [
        "a = torch.Tensor([1, 2, 3]).dtype"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "F22_FLpcPQX9"
      },
      "source": [
        "a"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7umnZhtaJMrU"
      },
      "source": [
        "#device = torch.device(\"cuda\")\n",
        "device = torch.device(\"cuda\")\n",
        "a = torch.Tensor([1, 2, 3]).to(device)  # Load on GPU\n",
        "b = torch.Tensor([4, 5, 6])             # Load on CPU"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5IPjfAPqPY4S"
      },
      "source": [
        "b"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OI4SgBWAJtIP"
      },
      "source": [
        "# This returns an error. Comment out before running the cell\n",
        "#a + b"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sF6y2gm3Jvh6"
      },
      "source": [
        "# GPU -> CPU\n",
        "a.detach().cpu() + b"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lbfmdJusJzvj"
      },
      "source": [
        "# PyTorch Tensor -> NumPy Array\n",
        "a.detach().cpu().numpy()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WzTE0N7nKc-A"
      },
      "source": [
        "# PyTorch Tensor -> Built-in List\n",
        "a.detach().cpu().tolist()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bbhFl7rJZLHj"
      },
      "source": [
        "## Logistic Regression/Multi-layer Perceptron with PyTorch\n",
        "\n",
        "Run the following code to conduct an experiment with `LogisticRegression`. Keep the default configuration for the first experiment. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3a6mnKfVWZW-"
      },
      "source": [
        "import random\n",
        "\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "from sklearn.datasets import load_digits\n",
        "from sklearn.metrics import accuracy_score\n",
        "from sklearn.model_selection import train_test_split\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "from torch import optim\n",
        "from torch.utils.data import TensorDataset, DataLoader"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qglOK-jCjdMf"
      },
      "source": [
        "### Logistic Regression\n",
        "\n",
        "Note that `forward()` does not have to use `softmax()` as it is implemented in the `CrossEntropy` loss function. Please see the [PyTorch documentation](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vlN08OqxZOSZ"
      },
      "source": [
        "class LogisticRegression(nn.Module):\n",
        "    def __init__(self,\n",
        "                 num_input,\n",
        "                 num_output):\n",
        "        super(LogisticRegression, self).__init__()\n",
        "        self.linear = nn.Linear(num_input, num_output)\n",
        "\n",
        "    def forward(self, X):\n",
        "        out = self.linear(X)\n",
        "        return out"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cc_NyNYYjY5k"
      },
      "source": [
        "### Experiment Code (Copy this block for the assignments)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7SgfxBj0NfBb"
      },
      "source": [
        "## Configurations ======\n",
        "n_epochs = 10\n",
        "batch_size = 16\n",
        "\n",
        "lr = 0.01\n",
        "momentum = 0.\n",
        "\n",
        "num_input = 64\n",
        "num_output = 10\n",
        "\n",
        "# Random Seeds\n",
        "torch.manual_seed(0)\n",
        "random.seed(0)\n",
        "np.random.seed(0)\n",
        "## ======================\n",
        "\n",
        "\n",
        "# GPU configuration\n",
        "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
        "\n",
        "# Load the handwritten digit dataset\n",
        "# https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_digits.html\n",
        "data = load_digits()\n",
        "X, y = data.data, data.target\n",
        "\n",
        "# Splint into 60% train, 20% valid, 20% test\n",
        "X_train, X_test, y_train, y_test = train_test_split(\n",
        "    X, y, test_size=0.2, random_state=1)\n",
        "\n",
        "X_train, X_valid, y_train, y_valid = train_test_split(\n",
        "    X_train, y_train, test_size=0.25, random_state=1)  # 0.25 x 0.8 = 0.2\n",
        "\n",
        "# NumPy array -> Torch tensor -> Dataset -> DataLoader\n",
        "# for the train, validation, test datasets\n",
        "dataset_train = TensorDataset(torch.Tensor(X_train),\n",
        "                              torch.LongTensor(y_train))\n",
        "dl_train = DataLoader(dataset_train,\n",
        "                      batch_size=batch_size,\n",
        "                      shuffle=True)\n",
        "\n",
        "dataset_valid = TensorDataset(torch.Tensor(X_valid),\n",
        "                              torch.LongTensor(y_valid))\n",
        "dl_valid = DataLoader(dataset_valid)\n",
        "\n",
        "dataset_test = TensorDataset(torch.Tensor(X_test),\n",
        "                              torch.LongTensor(y_test))\n",
        "dl_test = DataLoader(dataset_test)\n",
        "\n",
        "# Model, Optimzier, Loss function\n",
        "model = LogisticRegression(num_input=num_input,\n",
        "                           num_output=num_output).to(device)\n",
        "optimizer = optim.SGD(model.parameters(),\n",
        "                      lr=lr, momentum=momentum)\n",
        "\n",
        "loss_fn = nn.CrossEntropyLoss()\n",
        "\n",
        "# For each epoch\n",
        "eval_list = []\n",
        "for n in range(n_epochs):\n",
        "    print(\"Epoch {}\".format(n))\n",
        "    # Training\n",
        "    train_loss = 0.\n",
        "    train_pred_list = []\n",
        "    train_true_list = []\n",
        "    model.train()  # Switch to the training mode\n",
        "\n",
        "    # For each batch\n",
        "    for batch in dl_train:\n",
        "        optimizer.zero_grad()              # Initialize gradient information\n",
        "        X, y = batch\n",
        "        out = model(X.to(device))          # Call `forward()` function of the model\n",
        "        loss = loss_fn(out, y.to(device))  # Calculate loss \n",
        "        loss.backward()                    # Backpropagate the loss value\n",
        "        optimizer.step()                   # Update the parameters\n",
        "        \n",
        "        train_loss += loss.data.item() * batch_size\n",
        "        train_pred_list += out.argmax(1).detach().cpu().tolist()\n",
        "        train_true_list += y.detach().cpu().tolist()\n",
        "\n",
        "    train_loss /= len(dl_train)\n",
        "    train_acc = accuracy_score(train_true_list, train_pred_list)\n",
        "    print(\"    Training loss: {:.4f}\\t  Training acc: {:.4f}\".format(train_loss, train_acc))\n",
        "\n",
        "    # Validation\n",
        "    valid_loss = 0.\n",
        "    valid_pred_list = []\n",
        "    valid_true_list = []\n",
        "\n",
        "    model.eval()  # Switch to the evaluation mode\n",
        "    for batch in dl_valid:\n",
        "        X, y = batch\n",
        "        out = model(X.to(device))\n",
        "        loss = loss_fn(out, y.to(device))\n",
        "        valid_loss += loss.data.item() * batch_size\n",
        "        valid_pred_list.append(out.argmax(1).detach().cpu())\n",
        "        valid_true_list.append(y.detach().cpu())\n",
        "\n",
        "    valid_loss /= len(dl_valid)\n",
        "    valid_acc = accuracy_score(valid_true_list, valid_pred_list)\n",
        "    print(\"  Validation loss: {:.4f}\\tValidation acc: {:.4f}\".format(valid_loss, valid_acc))\n",
        "    # Store train/validation loss, accuracy values\n",
        "    eval_list.append([n, train_loss, train_acc, valid_loss, valid_acc])\n",
        "\n",
        "eval_df = pd.DataFrame(eval_list, columns=[\"epoch\", \"train_loss\", \"train_acc\",\n",
        "                                           \"valid_loss\", \"valid_acc\"])\n",
        "\n",
        "# Test\n",
        "model.eval()\n",
        "pred_list = []\n",
        "true_list = []\n",
        "for batch in dl_test:\n",
        "    X, y = batch\n",
        "    out = model(X.to(device))\n",
        "    pred = out.argmax().item()\n",
        "    pred_list.append(pred)\n",
        "    true_list.append(y.item())\n",
        "y_pred = np.array(pred_list)\n",
        "y_true = np.array(true_list)\n",
        "\n",
        "test_accuracy = accuracy_score(y_true, y_pred)\n",
        "print(\"\\nTest accuracy: {:.4f}\".format(test_accuracy))\n",
        "\n",
        "eval_df[[\"train_loss\", \"valid_loss\"]].plot()\n",
        "eval_df[[\"train_acc\", \"valid_acc\"]].plot()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8QqH41qkiWL_"
      },
      "source": [
        "# Please do NOT directly edit the code above but copy the code to blocks below"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "subQbMZRZKG5"
      },
      "source": [
        "# Assignments (due Fri 9/17)\n",
        "\n",
        "- Quiz 1. Complete the `MLP` class and run the same experiments with the MLP model. Use the default hidden layer size and the original random seed (i.e., `0`). Do you see better performance? How do `{train|valid}_loss` and `{train|valid}_acc` look like compared to those for `LogisticRegression`?\n",
        "\n",
        "- Quiz 2. Replace the logistic sigmoid function with ReLU in `MLP` (`MLPReLU`) and run the same experiment. Discuss the difference. \n",
        "\n",
        "- Quiz 3. Try smaller `learning_rate` (e.g., `0.001`) and larger `n_epochs` (e.g., `100`) for training (with `SGD` and `MLP`). Does it give more/less stable performance? Discuss why. \n",
        "\n",
        "- Quiz 4. Replace `SGD` with another optimizer `Adam` and run the same experiment (`n_epochs=10`). Look at [torch.optim](https://pytorch.org/docs/stable/optim.html) for the details of `Adam`. Discuss the resuls. \n",
        "\n",
        "- Quiz 5. Summarize the test accuracies of the experiments. Discuss the results. Which configuration is the best? Why? (Hint: Is this random effect? If you think so, how can you remove the randomness from the experiments?)\n",
        "\n",
        "\n",
        "After execution, keep the figures/results as they are and print the Colab notebook as PDF. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_M-gzPKthORN"
      },
      "source": [
        "## Quiz 1\n",
        "\n",
        "Complete the `MLP` class and run the same experiments with the MLP model. Use the default hidden layer size and the original random seed (i.e., `0`). Do you see better performance? How do `{train|valid}_loss` and `{train|valid}_acc` look like compared to those for `LogisticRegression`?\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EWWmMRbUZifD"
      },
      "source": [
        "class MLP(nn.Module):\n",
        "    def __init__(self,\n",
        "                 num_input,\n",
        "                 num_output,\n",
        "                 num_hidden=16):\n",
        "        super(MLP, self).__init__()\n",
        "        self.linear = nn.Linear(_, _)  ## COMPLETE CODE ##\n",
        "        self.sigmoid =                 ## COMPLETE CODE ##\n",
        "        self.hidden = nn.Linear(_, _)  ## COMPLETE CODE ##\n",
        "\n",
        "    def forward(self, X):\n",
        "        out = self.linear(X)\n",
        "        ## ADD 1 LINE HERE ##\n",
        "        ## ADD 1 LINE HERE ##\n",
        "        return out"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RaNH6GABh6mV"
      },
      "source": [
        "## Paste the experiment code here and modify it\n",
        "\n",
        "## Configurations ======\n",
        "n_epochs = 10\n",
        "batch_size = 16\n",
        "\n",
        "lr = 0.01\n",
        "momentum = 0.\n",
        "\n",
        "num_input = 64\n",
        "num_output = 10\n",
        "\n",
        "# Random Seeds\n",
        "torch.manual_seed(0)\n",
        "random.seed(0)\n",
        "np.random.seed(0)\n",
        "## ======================\n",
        "\n",
        "\n",
        "# GPU configuration\n",
        "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
        "\n",
        "# Load the handwritten digit dataset\n",
        "# https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_digits.html\n",
        "data = load_digits()\n",
        "X, y = data.data, data.target\n",
        "\n",
        "# Splint into 60% train, 20% valid, 20% test\n",
        "X_train, X_test, y_train, y_test = train_test_split(\n",
        "    X, y, test_size=0.2, random_state=1)\n",
        "\n",
        "X_train, X_valid, y_train, y_valid = train_test_split(\n",
        "    X_train, y_train, test_size=0.25, random_state=1)  # 0.25 x 0.8 = 0.2\n",
        "\n",
        "# NumPy array -> Torch tensor -> Dataset -> DataLoader\n",
        "# for the train, validation, test datasets\n",
        "dataset_train = TensorDataset(torch.Tensor(X_train),\n",
        "                              torch.LongTensor(y_train))\n",
        "dl_train = DataLoader(dataset_train,\n",
        "                      batch_size=batch_size,\n",
        "                      shuffle=True)\n",
        "\n",
        "dataset_valid = TensorDataset(torch.Tensor(X_valid),\n",
        "                              torch.LongTensor(y_valid))\n",
        "dl_valid = DataLoader(dataset_valid)\n",
        "\n",
        "dataset_test = TensorDataset(torch.Tensor(X_test),\n",
        "                              torch.LongTensor(y_test))\n",
        "dl_test = DataLoader(dataset_test)\n",
        "\n",
        "# Model, Optimzier, Loss function\n",
        "model = LogisticRegression(num_input=num_input,\n",
        "                           num_output=num_output).to(device)\n",
        "optimizer = optim.SGD(model.parameters(),\n",
        "                      lr=lr, momentum=momentum)\n",
        "\n",
        "loss_fn = nn.CrossEntropyLoss()\n",
        "\n",
        "# For each epoch\n",
        "eval_list = []\n",
        "for n in range(n_epochs):\n",
        "    print(\"Epoch {}\".format(n))\n",
        "    # Training\n",
        "    train_loss = 0.\n",
        "    train_pred_list = []\n",
        "    train_true_list = []\n",
        "    model.train()  # Switch to the training mode\n",
        "\n",
        "    # For each batch\n",
        "    for batch in dl_train:\n",
        "        optimizer.zero_grad()              # Initialize gradient information\n",
        "        X, y = batch\n",
        "        out = model(X.to(device))          # Call `forward()` function of the model\n",
        "        loss = loss_fn(out, y.to(device))  # Calculate loss \n",
        "        loss.backward()                    # Backpropagate the loss value\n",
        "        optimizer.step()                   # Update the parameters\n",
        "        \n",
        "        train_loss += loss.data.item() * batch_size\n",
        "        train_pred_list += out.argmax(1).detach().cpu().tolist()\n",
        "        train_true_list += y.detach().cpu().tolist()\n",
        "\n",
        "    train_loss /= len(dl_train)\n",
        "    train_acc = accuracy_score(train_true_list, train_pred_list)\n",
        "    print(\"    Training loss: {:.4f}\\t  Training acc: {:.4f}\".format(train_loss, train_acc))\n",
        "\n",
        "    # Validation\n",
        "    valid_loss = 0.\n",
        "    valid_pred_list = []\n",
        "    valid_true_list = []\n",
        "\n",
        "    model.eval()  # Switch to the evaluation mode\n",
        "    for batch in dl_valid:\n",
        "        X, y = batch\n",
        "        out = model(X.to(device))\n",
        "        loss = loss_fn(out, y.to(device))\n",
        "        valid_loss += loss.data.item() * batch_size\n",
        "        valid_pred_list.append(out.argmax(1).detach().cpu())\n",
        "        valid_true_list.append(y.detach().cpu())\n",
        "\n",
        "    valid_loss /= len(dl_valid)\n",
        "    valid_acc = accuracy_score(valid_true_list, valid_pred_list)\n",
        "    print(\"  Validation loss: {:.4f}\\tValidation acc: {:.4f}\".format(valid_loss, valid_acc))\n",
        "    # Store train/validation loss, accuracy values\n",
        "    eval_list.append([n, train_loss, train_acc, valid_loss, valid_acc])\n",
        "\n",
        "eval_df = pd.DataFrame(eval_list, columns=[\"epoch\", \"train_loss\", \"train_acc\",\n",
        "                                           \"valid_loss\", \"valid_acc\"])\n",
        "\n",
        "# Test\n",
        "model.eval()\n",
        "pred_list = []\n",
        "true_list = []\n",
        "for batch in dl_test:\n",
        "    X, y = batch\n",
        "    out = model(X.to(device))\n",
        "    pred = out.argmax().item()\n",
        "    pred_list.append(pred)\n",
        "    true_list.append(y.item())\n",
        "y_pred = np.array(pred_list)\n",
        "y_true = np.array(true_list)\n",
        "\n",
        "test_accuracy = accuracy_score(y_true, y_pred)\n",
        "print(\"\\nTest accuracy: {:.4f}\".format(test_accuracy))\n",
        "\n",
        "eval_df[[\"train_loss\", \"valid_loss\"]].plot()\n",
        "eval_df[[\"train_acc\", \"valid_acc\"]].plot()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tDI4BYpAiIdA"
      },
      "source": [
        "## Quiz 2\n",
        "\n",
        "Quiz 2. Replace the logistic sigmoid function with ReLU in `MLP` (`MLPReLU`) and run the same experiment. Discuss the difference. \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZsEIG_aOlC6S"
      },
      "source": [
        "class MLPReLU(nn.Module):\n",
        "    def __init__(self,\n",
        "                 num_input,\n",
        "                 num_output,\n",
        "                 num_hidden=16):\n",
        "        super(MLP, self).__init__()\n",
        "        self.linear = nn.Linear(_, _)  ## COMPLETE CODE ##\n",
        "        self.sigmoid =                 ## COMPLETE CODE ##\n",
        "        self.hidden = nn.Linear(_, _)  ## COMPLETE CODE ##\n",
        "\n",
        "    def forward(self, X):\n",
        "        out = self.linear(X)\n",
        "        ## ADD 1 LINE HERE ##\n",
        "        ## ADD 1 LINE HERE ##\n",
        "        return out"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "p7Yz91kViP4S"
      },
      "source": [
        "## Paste the experiment code here and modify it"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fp-nUD4viuOC"
      },
      "source": [
        "## Quiz 3\n",
        "\n",
        "Try smaller `learning_rate` (e.g., `0.001`) and larger `n_epochs` (e.g., `100`) for training (with `SGD` and `MLPReLU`). Does it give more/less stable performance? Discuss why. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IH152BxLi5yG"
      },
      "source": [
        "## Paste the experiment code here and modify it\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mz2r0pSxjFwP"
      },
      "source": [
        "## Quiz 4\n",
        "\n",
        "Quiz 4. Replace `SGD` with another optimizer `Adam` and run the same experiment (`n_epochs=10`). Look at [torch.optim](https://pytorch.org/docs/stable/optim.html) for the details of `Adam`. Discuss the resuls. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EjS29dCrjQXa"
      },
      "source": [
        "## Paste the experiment code here and modify it"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oUCp0RSBnyaq"
      },
      "source": [
        "## Quiz 5\n",
        "\n",
        "Summarize the test accuracies of the experiments. Discuss the results. Which configuration is the best? Why? (Hint: Is this random effect? If you think so, how can you remove the randomness from the experiments?)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9MXP4Oohn1ty"
      },
      "source": [
        "(Write your answer here)"
      ]
    }
  ]
}