alignment_exercise.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Pzh9ln5I_dPd"
   },
   "source": [
    "Before you run the code, make sure the runtime type is GPU (Runtime -> Change runtime type -> GPU)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "lE5lMjUdwB3D"
   },
   "outputs": [],
   "source": [
    "%%capture \n",
    "!pip install robustness\n",
    "!git clone https://github.com/MadryLab/AdvEx_Tutorial.git code\n",
    "!mv code/*.py .\n",
    "!wget http://people.csail.mit.edu/shibani/tutorial.zip\n",
    "!unzip tutorial\n",
    "!mv tutorial/* ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "exqGyhuo-hZx",
    "outputId": "91207db8-0c78-4337-9e5f-71145013912a"
   },
   "outputs": [],
   "source": [
    "!ls "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "rF1TJ1fIwHI0"
   },
   "outputs": [],
   "source": [
    "try: # set up path\n",
    "    import google.colab, sys, torch\n",
    "    if not torch.cuda.is_available():\n",
    "        print(\"Please change runtime type to include a GPU.\")  \n",
    "except:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 71
    },
    "colab_type": "code",
    "id": "5kuhxDcTv-Gg",
    "outputId": "cd164293-8e21-4a9d-b76f-1d976c6a1176"
   },
   "outputs": [],
   "source": [
    "# Import basic libraries needed for the exercise (numpy, matplotlib, and torch)\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from tqdm import tqdm\n",
    "import torch as ch\n",
    "import torchvision.transforms as transforms\n",
    "\n",
    "# We also use the robustness library (https://robustness.readthedocs.io/en/latest/) for some \n",
    "# convenient functionality.\n",
    "from robustness.tools.vis_tools import show_image_row\n",
    "\n",
    "import utils \n",
    "\n",
    "sns.set_style('darkgrid')\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "CbxLppqAv-Gk"
   },
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Sehuu91Vv-Gk"
   },
   "source": [
    "### Load datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "quwRUwTZv-Gl"
   },
   "source": [
    "For our experiments (except Ex. II), we will use the ImageNet dataset from the ILSVRC challenge. This is a 1000 class dataset, that has played an important role in developing and evaluating deep learning models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "_TXDcdRtv-Gl",
    "outputId": "69301f98-a81c-4227-f7b9-c0bb6f4ddbd6"
   },
   "outputs": [],
   "source": [
    "# Creater a dataset, and a loader to access it. In addition to the loader, we also need to obtain a\n",
    "# normalization function. This is because standard deep networks are typically trained \n",
    "# on normalized images, so we need to apply the same normalization during testing. Finally,\n",
    "# we also get a label map, that tells us what class a corresponding numeric value corresponds\n",
    "# to.\n",
    "in_dataset, in_loaders, normalization_function, label_map_IN = utils.load_dataset('imagenet',\n",
    "                                                                 batch_size=5,\n",
    "                                                                 num_workers=1)\n",
    "in_loader = in_loaders[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RGoBlcACv-Go"
   },
   "source": [
    "We can visualize some ImageNet samples, along with their labels, as follows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 167
    },
    "colab_type": "code",
    "id": "yzFCpxGxv-Go",
    "outputId": "a468533b-2a86-404d-9186-c87d6b12944f"
   },
   "outputs": [],
   "source": [
    "_, (img, targ) = next(enumerate(in_loader))\n",
    "\n",
    "show_image_row([img],\n",
    "              [\"ImageNet Images\"],\n",
    "              tlist=[[label_map_IN[int(t)].split(',')[0] for t in targ]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vJwUgnnfv-Gq"
   },
   "source": [
    "## Load model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5t5iyi-rv-Gr"
   },
   "source": [
    "Next, we need a model to play with! PyTorch provides access to a large range of pre-trained deep networks (for a full list, see <https://pytorch.org/docs/stable/torchvision/models.html>). For example, we can load a ResNet18 using the following code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 103,
     "referenced_widgets": [
      "cf3d338a3e0d4d6c9e8d47d23abf7c1a",
      "fb445702be25404fa29c10e7461bb0e2",
      "82d97a20caab4c7c83d41b9149be24a5",
      "7106747f5eac4286b49d9a49d33d64e0",
      "5443d05074074a7b8cdf6aaf22ae8bde",
      "a6e18480d09d45f0adfda751f5673406",
      "64c8c660ae09476a8b0c2839a8e323cc",
      "0d5cba86a8ba444395af9bb5016ce6c9"
     ]
    },
    "colab_type": "code",
    "id": "aP-A-NzCv-Gr",
    "outputId": "2900d15b-745e-48a9-b62f-132f0dba3a7c"
   },
   "outputs": [],
   "source": [
    "std_model = utils.load_model('resnet18')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fXKgaRWev-Gu"
   },
   "source": [
    "# Excercise I: Adversarial examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "WUCdz6hxv-Gu"
   },
   "source": [
    "Since their discovery, adversarial examples have been one of the most extensively studied phenomena in deep learning. Adversarial perturbations are *imperceptible* (non-random) perturbations that can be added to any input image so as to cause a standard (highly accurate) classifier to misclassify the modified input (or classify it as an adversarially chosen class). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "i6qrbV9Zv-Gu"
   },
   "source": [
    "*Finding adversarial examples:* The idea is pretty simple: given a target class (t), we want to find a perturbation ($\\delta'$) that when added to the input (x) maximizes the likelihood of the target class. At the same time, we want the perturbation to be small or lie within some pre-defined perturbation set: for example in a tiny L2 ball around the image. Basically, we want to find a $\\delta'$ such that\n",
    "\n",
    "$\\delta' = argmin_{||\\delta||_2 \\leq \\epsilon} L(x + \\delta, t; \\theta)$\n",
    "\n",
    "\n",
    "To find a perturbation that minimizes the objective (maximizes likelihood) while remaining in a bounded set, we use projected gradient descent PGD (see <https://arxiv.org/abs/1706.06083> for more). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "VZTrYEJJv-Gv"
   },
   "source": [
    "### Try it yourself! \n",
    "\n",
    "First choose a target class for every input. (Note that you have a batch of inputs, so you could try different targets for different inputs.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "hJ1nT-b2v-Gv",
    "outputId": "5d303326-9a1b-4cdf-c846-42c69b2676cc"
   },
   "outputs": [],
   "source": [
    "TARGET = 3\n",
    "\n",
    "print(f\"Target class: {label_map_IN[TARGET]}\")\n",
    "\n",
    "target_class = TARGET * ch.ones_like(targ)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Z7y0ujIYv-Gy"
   },
   "source": [
    "Next, there are a couple of parameters that you need to choose: \n",
    "1. eps: maximum size of the perturbation in terms of L2 norm. For e.g., eps=2 implies that $||\\delta||_2 \\leq 2$\n",
    "2. Nsteps: number of (projected) gradient descent to perform\n",
    "3. step_size: size of each step of (projected) gradient descent\n",
    "\n",
    "Try varying these parameters and see what happens.\n",
    "\n",
    "(You could also try implementing the PGD function yourself!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "hyo_OIjNv-Gy",
    "outputId": "b7615035-d827-4910-f928-a57aa6446b4d"
   },
   "outputs": [],
   "source": [
    "# Create adversarial examples\n",
    "adv_ex = utils.L2PGD(std_model, img, target_class, normalization_function,\n",
    "                       step_size=0.5, Nsteps=20, \n",
    "                       eps=1.25, targeted=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_ZZig2A7v-G0"
   },
   "source": [
    "### Evaluate model predictions at perturbed inputs\n",
    "\n",
    "To see if our attack was successful, we will now evaluate model predictions at the perturbed inputs. We would expect the predicted label to match the `target_class` used above, if we succeeded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "c_Su8zCDv-G0"
   },
   "outputs": [],
   "source": [
    "with ch.no_grad():\n",
    "    logits = utils.forward_pass(std_model, \n",
    "                                      adv_ex, \n",
    "                                      normalization_function)\n",
    "    pred_label = logits.argmax(dim=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "YtQVh9cVv-G2"
   },
   "source": [
    "### Visualize adversarial examples\n",
    "\n",
    "We now inspect the original (unperturbed) inputs (*top*), along with the corresponding adversarial examples (*bottom*). We also look at what the model predicts for each row of images.\n",
    "\n",
    "Does the attack succeed? Do the adversarial examples look different from the original inputs?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 315
    },
    "colab_type": "code",
    "id": "BkloMYl8v-G3",
    "outputId": "78dd2cc9-fae4-4d42-8c5c-34e1efd405e8"
   },
   "outputs": [],
   "source": [
    "show_image_row([img, adv_ex], \n",
    "               ['Original image', 'Adv. Example'],\n",
    "               tlist=[[label_map_IN[int(t)].split(',')[0] for t in label] \\\n",
    "                      for label in [targ, pred_label, targ]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "dBlZGgc2v-G5"
   },
   "source": [
    "# Excercise II: Are adversarial perturbations meaningless?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "w2Zb8bLNv-G6"
   },
   "source": [
    "In this experiment, we will revisit the cause underlying the brittleness of models to adversarial perturbations. In particular, we will examine if adversarial perturbations indeed correspond to meaningless sensitivities (or bugs) in the model.\n",
    "\n",
    "For computational efficiency, we will perform this experiment using linear classifiers trained on a binary subset of the CIFAR-10 dataset. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rjEt2zDtv-G6"
   },
   "source": [
    "Let's start by loading the dataset and looking at some of its samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 306,
     "referenced_widgets": [
      "025c326236cd48e48cfb5fc83403b8a7",
      "8e5da4ae9373457f823a7c7ec28dec8f",
      "d47a823649334878a19883d16c94b071",
      "6cbb663fb4cd48c5aaf1d2a0544d4e95",
      "60838b71f99e4445b92d53aa722d5687",
      "2c4c64e0da04428b8c74a7f55d0b3ca0",
      "cb153a59ff72473bac8b2f6dcaa29f57",
      "777111e1581c40e69acd4dc002f400fe"
     ]
    },
    "colab_type": "code",
    "id": "UH_menT8v-G7",
    "outputId": "555cbf0b-93ce-4e64-eca5-48eba58a7034"
   },
   "outputs": [],
   "source": [
    "binary_data = utils.load_binary_dataset(batch_size=100, num_workers=1, classes=[0, 3])\n",
    "\n",
    "im, targ = binary_data['train']\n",
    "show_image_row([im[:5]],\n",
    "               tlist=[[f\"Label={int(t)}\" for t in targ[:5]]],\n",
    "               fontsize=20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "d7sM-inKv-G9"
   },
   "source": [
    "### Training a linear classifier on this classification task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0sn4nv5Bv-G9"
   },
   "source": [
    "As you can see, the dataset contains two classes: cats and airplanes. We will now train a very basic linear classifier on the data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "oEvlU7kMv-G-",
    "outputId": "66b28074-d49c-4bbc-df12-efb35f7e4beb"
   },
   "outputs": [],
   "source": [
    "train_log, linear_net = utils.train_linear(binary_data, step_size=0.1, iterations=2000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "a65c92_nBNj1"
   },
   "source": [
    "If you instead prefer you could also load a pretrained model by setting `load_pretrained=True` in the code below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Ouz68dFCBFwU"
   },
   "outputs": [],
   "source": [
    "load_pretrained = False\n",
    "\n",
    "if load_pretrained:\n",
    "    Nfeatures = int(np.prod(binary_data['train'][0].shape[1:]))\n",
    "\n",
    "    linear_net = utils.Linear(Nfeatures=Nfeatures, Nclasses=2)\n",
    "    linear_net.load_state_dict(ch.load(\"./models/LinearCifarBinary.pt\"))\n",
    "    linear_net.eval()\n",
    "    linear_net = ch.nn.DataParallel(linear_net.cuda())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MkvRMptZv-HA"
   },
   "source": [
    "Let's take a look at some samples from the dataset, along with their labels and model predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 169
    },
    "colab_type": "code",
    "id": "HwNBPYOKv-HA",
    "outputId": "e8bda3ea-3b4e-4c9e-c097-ea16adf00db2"
   },
   "outputs": [],
   "source": [
    "preds = utils.get_predictions(im, linear_net)\n",
    "\n",
    "show_image_row([im[:5]],\n",
    "               tlist=[[f\"Label={int(t)} Pred={int(p)}\" \n",
    "                       for t, p in zip(targ[:5], preds[:10])]],\n",
    "               fontsize=16)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "PhVA9m3pv-HC"
   },
   "source": [
    "### Using adversarial examples to train models\n",
    "\n",
    "Note that adversarial examples correspond to adding a non-random perturbation to a given input data point. Thus adversarial perturbations modify input features, albeit in an imperceptible way.\n",
    "\n",
    "So what features do these perturbations modify? Do they just exploit meaningless sensitivities (or bugs) of the models? What happens if we train a new model solely on adversarial examples?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "lQmK4ijzv-HC"
   },
   "source": [
    "We will now construct a *training* dataset made of adversarial examples. Specifically, we will:\n",
    "\n",
    "Step 1. Add adversarial perturbations to all the training set images to fool the linear classifier into flipping its prediction (i.e., classify cats as `0` and airplanes as `1`).\n",
    "\n",
    "Step 2. We will now take all the successful adversarial examples (i.e., data points that were originally classified correctly by the model, but now after they have been adversarially perturbed) and use these to make a new dataset. The image labels in this dataset will be the labels *predicted* by the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "SM-tLZVvv-HD",
    "outputId": "e5b5d983-c639-4b25-cff7-0b8ee39c950d"
   },
   "outputs": [],
   "source": [
    "im_adv = utils.L2PGD(linear_net, im, targ, None,\n",
    "                       step_size=0.1, Nsteps=20, \n",
    "                       eps=1, targeted=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fgPWwBqBBdTq"
   },
   "source": [
    "Let's look at how many examples we managed to fool the network on and examine some of these samples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 351
    },
    "colab_type": "code",
    "id": "JE-fEEY8v-HF",
    "outputId": "72b3564c-0a15-49a6-b66f-9fd7d71a01a9"
   },
   "outputs": [],
   "source": [
    "preds_adv = utils.get_predictions(im_adv, linear_net)\n",
    "print(\"% examples on which model is fooled:\",\n",
    "      f\"{100 * ch.mean(preds_adv.cpu().eq(targ).float()).item():.2f} \\n\")\n",
    "\n",
    "idx = np.where(np.logical_and(preds.cpu() == targ, preds_adv.cpu() != targ))[0]\n",
    "np.random.shuffle(idx)\n",
    "\n",
    "show_image_row([im[idx[:5]], im_adv[idx[:5]]],\n",
    "               ylist=[\"Original\", \"Adv. example\"],\n",
    "               tlist=[[f\"Label={int(t)}, Pred={int(p)}\" \n",
    "                       for t, p in zip(targ[idx[:5]], preds[idx[:5]])],\n",
    "                      [f\"Label={int(t)}, Pred={int(p)}\" \n",
    "                       for t, p in zip(targ[idx[:5]], preds_adv[idx[:5]])]],\n",
    "               fontsize=16)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Rtb5qcRov-HH"
   },
   "source": [
    "Note that in the original data, airplanes were labeled as class `0` and cats were labeled as class `1`. However for the adversarial examples, the model predicts the opposite/incorrect label.\n",
    "\n",
    "We will now train a *new* model on these adversarial examples. Note crucially, that the data points are labeled based on the *predicted label* and hence the labels are flipped w.r.t. the original train/test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "qxGz4FCnv-HI",
    "outputId": "795e2a72-c19f-4d1a-87aa-b95093501ac8"
   },
   "outputs": [],
   "source": [
    "binary_data_adv = {}\n",
    "# Training data not consists solely of adv. examples which fooled the model \n",
    "# and their \"incorrect\" labels (airplanes -> 1 and cats -> 0)\n",
    "binary_data_adv['train'] = (im_adv[idx].cpu(), preds_adv[idx].cpu())\n",
    "# Test data remains the same (i.e., airplanes -> 0 and cats -> 1)  \n",
    "binary_data_adv['test'] = binary_data['test']\n",
    "\n",
    "# The train set size is smaller because we only choose the training samples we are able to fool the model on\n",
    "print(binary_data_adv['train'][0].shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "NiG2If0Uv-HJ"
   },
   "source": [
    "Let's look again at the training and test data\n",
    "* If you, as a human, were trained on the samples in the second row in the figure above what mapping would you learn between [cats, airplanes] and labels [0, 1]?\n",
    "* What would your accuracy on the original (unmodified) test set be?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 317
    },
    "colab_type": "code",
    "id": "VW-rTYdAv-HK",
    "outputId": "b1facaa9-8497-423b-b163-3db3bf2d91fb"
   },
   "outputs": [],
   "source": [
    "show_image_row([binary_data_adv['train'][0][:5], binary_data_adv['test'][0][:5]],\n",
    "               ylist=[\"Train\", \"Test\"],\n",
    "               tlist=[[f\"Label={int(t)}\" for t in binary_data_adv['train'][1][:5]],\n",
    "                      [f\"Label={int(t)}\" for t in binary_data_adv['test'][1][:5]]],\n",
    "               fontsize=16)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MuCfp1Q-Bv_k"
   },
   "source": [
    "We will now train a linear classifier from scratch on this mislabelled dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "I31mrfTJv-HM",
    "outputId": "c61be53c-08a0-4cea-8268-8dd2cdc17f6c"
   },
   "outputs": [],
   "source": [
    "train_log_adv, adv_net = utils.train_linear(binary_data_adv, step_size=0.1, iterations=3000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "TIgfSZ9Gv-HN"
   },
   "source": [
    "The model still gets > 70% accuracy on the original, unmodified test set, despite being trained on an entirely mislabeled training set!\n",
    "\n",
    "Let's look at the predictions of this model to be sure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 317
    },
    "colab_type": "code",
    "id": "ND5ySR1tv-HO",
    "outputId": "ddbba98d-edec-44e3-efc7-1daf51e15bf8"
   },
   "outputs": [],
   "source": [
    "preds = utils.get_predictions(binary_data_adv['test'][0], linear_net)\n",
    "\n",
    "\n",
    "show_image_row([binary_data_adv['train'][0][:5], binary_data_adv['test'][0][:5]],\n",
    "               ylist=[\"Train\", \"Test\"],\n",
    "               tlist=[[f\"Label={int(t)}\" for t in binary_data_adv['train'][1][:5]],\n",
    "                      [f\"Label={int(t)}, Pred={int(p)}\" \n",
    "                       for t, p in zip(binary_data_adv['test'][1][:5], preds[:5])]],\n",
    "               fontsize=16)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "a09lhoMrv-HP"
   },
   "source": [
    "So how did this happen? \n",
    "\n",
    "Note that all the human meaningful features, which we refer to as robust features, in these images point to the incorrect label (e.g., wings -> 1 and ears -> 0). Thus, a human trained on the dataset above would get 0% accuracy on the test set.\n",
    "\n",
    "Since it is not possible to get non-trivial accuracy on the test set based on robust features, this must be due to the (imperceptible) features we introduced via adversarial perturbations. For instance, when we added adversarial perturbations to a cat image to make the first linear classifier think it was a plane, we must have added features that actually generalize to planes on the test set. \n",
    "\n",
    "Thus, adversarial examples do not correspond just to meaningless sensitivies but to well-generalizing features. We see this phenomenon occur on state-of-the-art deep nets and on multi-class datasets such as CIFAR or ImageNet. You could try to reproduce this effect there as a follow-up exercise!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "cW3wybZwv-HQ"
   },
   "source": [
    "# Excercise III: Gradients as model interpretations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "-VCRtzglv-HQ"
   },
   "source": [
    "So far, we saw that standard models rely on non-robust features for part of their performance. We will now explore how this dependence affects other properties of standard models, specifically model interpretability.\n",
    "\n",
    "For this, we will begin by looking at one of the most natural interpretations: gradient-based saliency maps. These maps highlight which input features (pixels) the model prediction is sensitive to.\n",
    "\n",
    "(In these experiments, we will go back to the ImageNet-trained ResNet model from Ex. I.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "lAiexsZdv-HQ"
   },
   "source": [
    "### Compute and visualize gradient"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "N4Oj1pKzB-JO"
   },
   "outputs": [],
   "source": [
    "_, (img, targ) = next(enumerate(in_loader))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 315
    },
    "colab_type": "code",
    "id": "23joydOev-HR",
    "outputId": "e68a9066-912a-4fb0-fbe3-bf59312d5a2c"
   },
   "outputs": [],
   "source": [
    "# We compute the gradient of the loss, with respect to the input. For every image pixel,\n",
    "# the gradient tells us how the loss changes if we vary that pixel slightly.\n",
    "\n",
    "grad, _ = utils.get_gradient(std_model, img, targ, normalization_function)\n",
    "\n",
    "# We can then visualize the original image, along with the gradient. Note that the gradient may\n",
    "# not lie within the valid pixel range ([0, 1]), so we need to rescale it using the \n",
    "# `visualize_gradient` function.\n",
    "\n",
    "show_image_row([img, utils.visualize_gradient(grad)],\n",
    "              [\"Original Image\", \"Gradient\"],\n",
    "              tlist=[[label_map_IN[int(t)].split(',')[0] for t in targ],\n",
    "                     [\"\" for _ in targ]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qzQVIFBzv-HS"
   },
   "source": [
    "The gradients of standard models look quite noisy and seem rather hard to interpret. Why might this be the case? Could it have something to do with non-robust features?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "I2J3deT0v-HT"
   },
   "source": [
    "# Exercise IV:  Try SmoothGrad and visualize the interpretations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jGw4-Wnbv-HT"
   },
   "source": [
    "Fill in the following skeleton to implement SmoothGrad.\n",
    "\n",
    "```\n",
    "def smooth_grad(mod, im, targ, normalization, Nsamples, stdev):\n",
    "    it = tqdm(enumerate(range(Nsamples)), total=Nsamples)\n",
    "    total_grad = 0\n",
    "    for _, n in it:\n",
    "        ...\n",
    "        grad, _ = utils.get_gradient(mod, noised_im, targ, normalization)\n",
    "        total_grad += grad\n",
    "    return total_grad / Nsamples\n",
    "```\n",
    "    \n",
    "Then, try using SmoothGrad to interpret a standard model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "OqzO9wmcv-HU"
   },
   "outputs": [],
   "source": [
    "def smooth_grad(mod, im, targ, normalization,\n",
    "                Nsamples, stdev):\n",
    "    # Instead of taking the gradient of a single image, we will take gradients\n",
    "    # at a bunch of neighborhood points and average their gradients.\n",
    "    \n",
    "    it = tqdm(range(Nsamples), total=Nsamples)\n",
    "\n",
    "    total_grad = 0\n",
    "    for _ in it:\n",
    "        pass # FILL THIS IN\n",
    "    \n",
    "    # Return average gradient\n",
    "    return total_grad / Nsamples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "8ciBezL-v-HW",
    "outputId": "a8a5a68c-75ca-46af-c474-0b22473e1740"
   },
   "outputs": [],
   "source": [
    "sgrad = smooth_grad(std_model, img, targ, normalization_function,\n",
    "                    100, 0.3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 315
    },
    "colab_type": "code",
    "id": "Nu0XvNq-v-HX",
    "outputId": "769fd756-e587-481a-cf0c-b2f2bb55f776"
   },
   "outputs": [],
   "source": [
    "# We once again use the `visualize_gradient` helper to make the SmoothGrad suitable for \n",
    "# visualization.\n",
    "\n",
    "show_image_row([img, utils.visualize_gradient(sgrad)],\n",
    "              [\"Original Image\", \"SmoothGrad\"],\n",
    "               tlist=[[label_map_IN[int(t)].split(',')[0] for t in targ],\n",
    "                     [\"\" for _ in targ]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "f3u_2sptv-HZ"
   },
   "source": [
    "Explanations based on SmoothGrad align much better with features that we humans might use to make predictions. But what did we actually fix in smoothing the gradients? Were vanilla gradients just overly sensitive and noisy? Or did we maybe mask some  features that the models actually rely on to get cleaner interpretations?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0nzUoG-Lv-HZ"
   },
   "source": [
    "# Excercise V: Playing with robust models\n",
    "\n",
    "The existance of adversarial examples has also prompted a large body of research to build models that are robust to these perturbations, i.e., so-called *robust models*. One approach to get a robust model is to train against the PGD adversary: instead of minimizing the loss over training examples, we minimize the loss against adversarially perturbed training samples (obtained using PGD). We will now take a closer look at robust models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mLma7ET2v-Ha"
   },
   "source": [
    "### Loading a robust model\n",
    "\n",
    "For our study today, we will use a pre-trained robust model. We trained this model (ResNet50) on a 9-class subset of the ImageNet dataset. (Developing good robust models for the more complex 1000 class version is still an active area of research.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "colab_type": "code",
    "id": "wg7rUUDCv-Ha",
    "outputId": "9e4dffd4-e9e1-4f47-b503-5a18e05f1940"
   },
   "outputs": [],
   "source": [
    "# Load the \"Restricted\" ImageNet dataset\n",
    "restricted_imagenet_ds, rin_loaders, normalization_function, label_map_RIN = \\\n",
    "            utils.load_dataset('restricted_imagenet', batch_size=5, num_workers=1)\n",
    "\n",
    "rin_loader = rin_loaders[1]    \n",
    "# Load a pre-trained robust model\n",
    "#robust_model = utils.load_model('robust', restricted_imagenet_ds)\n",
    "robust_model = utils.load_model('robust', restricted_imagenet_ds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ANuBO_aAv-Hc"
   },
   "source": [
    "### Can adversarial examples fool a robust model?\n",
    "\n",
    "We can now try to fool the robust model using the same procedure as before. Does it succeed? Try varying the attack parameters and see what happens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 349
    },
    "colab_type": "code",
    "id": "p5KJBoDCv-Hc",
    "outputId": "d048b67e-68bb-4c0d-f0ae-6ecafbbf18b9"
   },
   "outputs": [],
   "source": [
    "# Load images from the Restricted ImageNet dataset\n",
    "_, (img, targ) = next(enumerate(rin_loader))\n",
    "\n",
    "# Then we choose a target label for the attack.\n",
    "TARGET = 3\n",
    "\n",
    "print(f\"Target class: {label_map_RIN[TARGET]}\")\n",
    "\n",
    "target_class = TARGET * ch.ones_like(targ)\n",
    "\n",
    "# Create adversarial examples\n",
    "adv_ex = utils.L2PGD(robust_model, img, target_class, normalization_function,\n",
    "                       step_size=0.5, Nsteps=20, eps=1.25, targeted=True)\n",
    "\n",
    "# Evaluate model predictions\n",
    "with ch.no_grad():\n",
    "    logit = utils.forward_pass(robust_model, adv_ex, normalization_function)\n",
    "    pred_label = logit.argmax(dim=1)\n",
    "\n",
    "# Visualize adversarial examples\n",
    "\n",
    "show_image_row([img, adv_ex], \n",
    "               ['Original image', 'Adv. Example'],\n",
    "               tlist=[[label_map_RIN[int(t)].split(',')[0] for t in label] \\\n",
    "                      for label in [targ, pred_label]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HrUpVbgMv-He"
   },
   "source": [
    "### Changing the prediction of a robust model\n",
    "\n",
    "We know that robust models are not easily fooled by adversarial examples. This tells us that one cannot change the prediction of a robust model using imperceptible L2 perturbations to the input (in contrast to standard models). How can we then modify the input to make the robust model predict a different class?\n",
    "\n",
    "Try creating adversarial examples as before, but with a larger eps. Our hope is that by increasing the size of the perturbation set, we can find a perturbation that actually causes the model to change its prediction? What do the perturbed inputs, i.e., \"*large epsilon adversarial examples*\" look like?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "E_kw0gsov-He",
    "outputId": "456eba7f-c602-4a93-8fb1-3006e5a1aa69"
   },
   "outputs": [],
   "source": [
    "TARGET = 5\n",
    "\n",
    "print(f\"Target class: {label_map_RIN[TARGET]}\")\n",
    "\n",
    "target_class = TARGET * ch.ones_like(targ)\n",
    "\n",
    "im_targ = utils.L2PGD(robust_model, img, target_class, normalization_function,\n",
    "                        step_size=5, Nsteps=20, eps=100, targeted=True)\n",
    "\n",
    "# Evaluate model predictions\n",
    "with ch.no_grad():\n",
    "    logit = utils.forward_pass(robust_model, im_targ, normalization_function)\n",
    "    pred_label = logit.argmax(dim=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 315
    },
    "colab_type": "code",
    "id": "tNcfI5Wmv-Hg",
    "outputId": "dc6782a6-a282-462a-f44a-1d6fc86b1c62"
   },
   "outputs": [],
   "source": [
    "show_image_row([img, im_targ],\n",
    "              ['Original image', 'Large eps \\n adv. example'],\n",
    "               tlist=[[label_map_RIN[int(t)].split(',')[0] for t in label] \\\n",
    "                      for label in [targ, pred_label]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "z3RyX8ngv-Hh"
   },
   "source": [
    "# Excercise VI: Interpretations for robust models\n",
    "\n",
    "Based on the previous experiment, we know that, for robust models, (a) imperceptible input changes do not change the prediction and (b) to change the prediction, we actually need to change \"salient image features\".\n",
    "\n",
    "Does this mean that the features that robust models rely on are more human-aligned in a sense?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1CjltHksv-Hi"
   },
   "source": [
    "### VI.I Let's start by looking at their gradients.\n",
    "\n",
    "What do the gradients of robust models look like? How do they compare to the gradients of a standard model and the output of SmoothGrad?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 315
    },
    "colab_type": "code",
    "id": "ptvCWsNxv-Hi",
    "outputId": "0e0c8e2c-38f3-4b0e-fe5c-75f16a4a6797"
   },
   "outputs": [],
   "source": [
    "# Get gradient of the loss with respect to the input\n",
    "grad_rob, _ = utils.get_gradient(robust_model, img, targ, normalization_function)\n",
    "\n",
    "# Visualize gradient\n",
    "show_image_row([img, utils.visualize_gradient(grad_rob)],\n",
    "              [\"Original Image\", \"Gradient\"],\n",
    "              tlist=[[label_map_RIN[int(t)].split(',')[0] for t in targ],\n",
    "                     [\"\" for _ in targ]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6Gy1pQuWv-Hk"
   },
   "source": [
    "### VI.2 Feature Visualization\n",
    "\n",
    "Another popular interpretability technique is known as feature visualization. Here, the goal is to find an input that maximizes a feature (a particular neuron in the deep network), instead of just trying to maximize the loss (as we did before with gradients).\n",
    "\n",
    "You could now try to implement feature visualization yourself. For instance, the following function gives you, for specific inputs, the model's feature vector (the layer before the final linear classifier). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "dciqUDFrv-Hk",
    "outputId": "1bfcb595-ee73-48d7-bbea-8b3d476b9329"
   },
   "outputs": [],
   "source": [
    "# Getting the feature representation from the model\n",
    "with ch.no_grad():\n",
    "    feats = utils.get_features(robust_model, img, normalization_function)\n",
    "    print(f\"Dimensions of the feature vector: {feats.shape[1]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "CzWeK3aDv-Hl"
   },
   "source": [
    "#### Implement a loss function to perform feature visualization\n",
    "\n",
    "Fill in the skeleton below to create a feature visualization loss function. Our goal is to maximize the `feature_number` coordinate of the feature vector.\n",
    "\n",
    "```\n",
    "def feature_maximization_loss(mod, im, feature_number, normalization_function):\n",
    "    feature_vector = utils.get_features(mod, im, normalization) # Get features for input\n",
    "    relevant_coordinate = ch.gather(feature_vector, 1, feature_number[:, None]) \n",
    "    loss = ?\n",
    "    ...\n",
    "    return loss\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-DC5InP7v-Hm"
   },
   "outputs": [],
   "source": [
    "# Feature visualization loss: Try to find an input that maximizes a specific feature\n",
    "\n",
    "def feature_maximization_loss(mod, im, feature_number, normalization):\n",
    "    # Get feature vector for inputs\n",
    "    fr = utils.get_features(mod, im, normalization)\n",
    "    # We will maximize the `targ` coordinate of the feature vector for every input\n",
    "    loss = 'FILL THIS IN'\n",
    "    return loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6uLITJmov-Ho"
   },
   "source": [
    "#### Visualize the results of feature visualization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Q-nvE0bNv-Hp"
   },
   "source": [
    "You can then supply the `feature_maximization_loss` to the `custom_loss` argument input in `helpers.L2PGD`, and maximize it using the following snippet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "insyk95_v-Hp"
   },
   "outputs": [],
   "source": [
    "# Chose a feature to visualize\n",
    "FEATURE = 200 # should be less than the dimension from before\n",
    "\n",
    "target_feature = FEATURE * ch.ones_like(targ)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "L8sbOqSwv-Hr",
    "outputId": "c234a7c1-4407-43bf-aca3-ad2934698424"
   },
   "outputs": [],
   "source": [
    "# Maximize feature \n",
    "im_f = utils.L2PGD(robust_model, img, target_feature, normalization_function,\n",
    "                              step_size=5, Nsteps=20, eps=1000, \n",
    "                              custom_loss=feature_maximization_loss, \n",
    "                              targeted=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 311
    },
    "colab_type": "code",
    "id": "u6q-2yrLv-Ht",
    "outputId": "f50816dd-922d-4311-c36a-b88d6d768da7"
   },
   "outputs": [],
   "source": [
    "# Visualize results\n",
    "show_image_row([img, im_f],\n",
    "               [\"Original Image\", f\"Maximize Feature #{FEATURE}\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qjawi1KTv-Hu"
   },
   "source": [
    "#### Try the same for a standard model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "ydPt6Ldnv-Hv",
    "outputId": "f385e437-1e29-4f61-b307-e5f8da127d86"
   },
   "outputs": [],
   "source": [
    "# Load image-label pair from ImageNet\n",
    "_, (img, targ) = next(enumerate(in_loader))\n",
    "\n",
    "TARGET = 100 \n",
    "target_feature = TARGET * ch.ones_like(targ)\n",
    "im_f = utils.L2PGD(std_model, img, target_feature, normalization_function,\n",
    "                              step_size=5, Nsteps=20, eps=1000, \n",
    "                              custom_loss=feature_maximization_loss, \n",
    "                              targeted=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 311
    },
    "colab_type": "code",
    "id": "q_2T3HVjv-Hx",
    "outputId": "254a6333-f5e6-49d0-b339-12ad5dca76c3"
   },
   "outputs": [],
   "source": [
    "show_image_row([img, im_f],\n",
    "               [\"Original Image\", f\"Maximize Feature #{FEATURE}\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "OSrj6BcGv-Hz"
   },
   "source": [
    "# Bonus Excercise I: Try feature visualization for robust models starting from noise rather than images\n",
    "\n",
    "What if we try feature visualization starting from noise?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "rmPP9C_Yv-Hz",
    "outputId": "e500c099-f790-4f9a-a4ac-1c5d1ed480a0"
   },
   "outputs": [],
   "source": [
    "# Create a \"noise\" image\n",
    "noise_img = ch.clamp(ch.randn_like(img) + 0.5, 0, 1)\n",
    "\n",
    "FEATURE = 201\n",
    "target_feature = FEATURE * ch.ones_like(targ)\n",
    "im_f = utils.L2PGD(robust_model, noise_img, target_feature, normalization_function,\n",
    "                     step_size=5, Nsteps=200, eps=1000, \n",
    "                     custom_loss=feature_maximization_loss, \n",
    "                     targeted=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 311
    },
    "colab_type": "code",
    "id": "uY4IiY8Hv-H2",
    "outputId": "591fb720-94ad-4439-b0ec-62903c6b55c2"
   },
   "outputs": [],
   "source": [
    "show_image_row([noise_img, im_f],\n",
    "                [\"Original Image\", f\"Maximize Feature #{FEATURE}\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "XCxBh4Utv-H4"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "name": "alignment_solution.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "025c326236cd48e48cfb5fc83403b8a7": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_d47a823649334878a19883d16c94b071",
       "IPY_MODEL_6cbb663fb4cd48c5aaf1d2a0544d4e95"
      ],
      "layout": "IPY_MODEL_8e5da4ae9373457f823a7c7ec28dec8f"
     }
    },
    "0d5cba86a8ba444395af9bb5016ce6c9": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "2c4c64e0da04428b8c74a7f55d0b3ca0": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "5443d05074074a7b8cdf6aaf22ae8bde": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "60838b71f99e4445b92d53aa722d5687": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "64c8c660ae09476a8b0c2839a8e323cc": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "6cbb663fb4cd48c5aaf1d2a0544d4e95": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_777111e1581c40e69acd4dc002f400fe",
      "placeholder": "​",
      "style": "IPY_MODEL_cb153a59ff72473bac8b2f6dcaa29f57",
      "value": " 170500096/? [00:09&lt;00:00, 17086638.92it/s]"
     }
    },
    "7106747f5eac4286b49d9a49d33d64e0": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_0d5cba86a8ba444395af9bb5016ce6c9",
      "placeholder": "​",
      "style": "IPY_MODEL_64c8c660ae09476a8b0c2839a8e323cc",
      "value": " 44.7M/44.7M [00:14&lt;00:00, 3.20MB/s]"
     }
    },
    "777111e1581c40e69acd4dc002f400fe": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "82d97a20caab4c7c83d41b9149be24a5": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_a6e18480d09d45f0adfda751f5673406",
      "max": 46827520,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_5443d05074074a7b8cdf6aaf22ae8bde",
      "value": 46827520
     }
    },
    "8e5da4ae9373457f823a7c7ec28dec8f": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "a6e18480d09d45f0adfda751f5673406": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "cb153a59ff72473bac8b2f6dcaa29f57": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "cf3d338a3e0d4d6c9e8d47d23abf7c1a": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_82d97a20caab4c7c83d41b9149be24a5",
       "IPY_MODEL_7106747f5eac4286b49d9a49d33d64e0"
      ],
      "layout": "IPY_MODEL_fb445702be25404fa29c10e7461bb0e2"
     }
    },
    "d47a823649334878a19883d16c94b071": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_2c4c64e0da04428b8c74a7f55d0b3ca0",
      "max": 1,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_60838b71f99e4445b92d53aa722d5687",
      "value": 1
     }
    },
    "fb445702be25404fa29c10e7461bb0e2": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}