HW4_MINA_1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "b5be520108ae7339c1d9573be0c7df10",
     "grade": false,
     "grade_id": "cell-d81b4e391e5f1b86",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "#  ECG Data Classification with MINA\n",
    "\n",
    "## Overview\n",
    "\n",
    "\n",
    "In this section, you will implement an advanced CNN+RNN model with attention mechanism to classify ECG recordings. Specifically, we face a binary classification problem, and the goal is to distinguish atrial fibrillation (AF), an alternative rhythm, from the normal sinus rhythm. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "22 EXERCISES"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-03-03T04:56:12.824696Z",
     "start_time": "2022-03-03T04:56:12.145672Z"
    },
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "d06faf5beb0615e46296e582d172c5c8",
     "grade": false,
     "grade_id": "cell-1fdbbeceab82d1de",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import random\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "# set seed\n",
    "seed = 24\n",
    "random.seed(seed)\n",
    "np.random.seed(seed)\n",
    "torch.manual_seed(seed)\n",
    "os.environ[\"PYTHONHASHSEED\"] = str(seed)\n",
    "\n",
    "# define data path\n",
    "DATA_PATH = \"../HW4_MINA-lib/data/\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "d60f978702ff85b35901beee24820925",
     "grade": false,
     "grade_id": "cell-fc347cd1779b8abc",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "## 1 ECG Data Data [10 points]\n",
    "\n",
    "We will be using a fraction of the data in the public [Physionet 2017 Challenge](https://physionet.org/content/challenge-2017/1.0.0/). More details can be found in the link.\n",
    "\n",
    "ECG recordings were sampled at 300Hz, and for the purpose of this task, the data we use is separated into 10-second-segments. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "61dd9478fd3c75feb5cf1d1cdb65c81b",
     "grade": false,
     "grade_id": "cell-e2da856b9e4cecf5",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "### 1.1 Preprocessing\n",
    "\n",
    "Because the preprocessing of the data requires a tremendous amount of memory and time, for the sake of this homework, the data has already been preprocessed. \n",
    "\n",
    "Specifically, for each raw data (an ECG recording sampled at 300Hz), we did the following:\n",
    "1. split the dataset into training/validation/test sets with a ratio of [placeholder]\n",
    "2. for each recording, we normalize the data to have a mean of 0 and a standard deviation of 1\n",
    "3. slide and cut the recording into overlapping 10-second-segments (stride = $\\frac{5}{3}$ second for class 0, and $\\frac{5}{30}$ second for class 1 to oversample).\n",
    "4. use FIR bandpass filter to transform the data from 1 channel to 4 channels.\n",
    "\n",
    "\n",
    "The last step of the data preprocessing is computing the knowledge. As we can see below, the AF signals exhibit different patterns at different levels. We computed knoledge features at different levels to guide the attention mechanism. More details are in Section 2.\n",
    "![Beat/Rhythm/Frequency](img/Data.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "000faeda9838d1094622aa20eea463c5",
     "grade": false,
     "grade_id": "cell-2f2ef91398b89a49",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "### 1.2 Load the Data [10 points]\n",
    "\n",
    "Due to the resource constraints, the data and knowledge features have already been computed. Let's load them below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-03-03T04:56:13.173896Z",
     "start_time": "2022-03-03T04:56:12.826845Z"
    },
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "675a556174ce29fce4ae48ed650f8b0a",
     "grade": false,
     "grade_id": "cell-683bbb341b249aa4",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    },
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "train_dict = pd.read_pickle(os.path.join(DATA_PATH, 'train.pkl'))\n",
    "test_dict = pd.read_pickle(os.path.join(DATA_PATH, 'test.pkl'))\n",
    "\n",
    "print(f\"There are {len(train_dict['Y'])} training data, {len(test_dict['Y'])} test data\")\n",
    "print(f\"Shape of X: {train_dict['X'][:, 0,:].shape} = (#channels, n)\")\n",
    "print(f\"Shape of beat feature: {train_dict['K_beat'][:, 0, :].shape} = (#channels, n)\")\n",
    "print(f\"Shape of rhythm feature: {train_dict['K_rhythm'][:, 0, :].shape} = (#channels, M)\")\n",
    "print(f\"Shape of frequency feature: {train_dict['K_freq'][:, 0, :].shape} = (#channels, 1)\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_dict = pd.read_pickle(os.path.join(DATA_PATH, 'train.pkl'))\n",
    "test_dict = pd.read_pickle(os.path.join(DATA_PATH, 'test.pkl'))\n",
    "\n",
    "print(f\"There are {len(train_dict['Y'])} training samples \\n \\\n",
    "ie: len(train_dict['Y'] = {len(train_dict['Y'])} training samples \\n \\\n",
    "There are {len(test_dict['Y'])} test samples \\n \\\n",
    "ie: len(test_dict['Y'] = {len(test_dict['Y'])}\")\n",
    "print(f\"Shape of X, \\n \\\n",
    "ie: train_dict['X'][:, 0,:]: {train_dict['X'][:, 0,:].shape} = (#channels, n)\")\n",
    "print(f\"Shape of beat feature, \\n \\\n",
    "ie: train_dict['K_beat'][:, 0, :]: {train_dict['K_beat'][:, 0, :].shape} = (#channels, n)\")\n",
    "print(f\"Shape of rhythm feature, \\n \\\n",
    "ie: train_dict['K_rhythm'][:, 0, :]: {train_dict['K_rhythm'][:, 0, :].shape} = (#channels, M)\")\n",
    "print(f\"Shape of frequency feature: \\n \\\n",
    "ie: train_dict['K_freq'][:, 0, :]: {train_dict['K_freq'][:, 0, :].shape} = (#channels, 1)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "5bcdedc741ad7c5b4b92c0b81aef35ed",
     "grade": false,
     "grade_id": "cell-d395e587202afcbf",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "You will need to define a ECGDataset class, and then define the DataLoader as well. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# [x**2 for x in range(10)] # [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]\n",
    "[(x, y) for x in [1,2,3] for y in [3,1,4] if x != y]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = torch.ones(1,0,1)\n",
    "b = torch.ones(1,1,1)\n",
    "c = torch.cat((a,b), dim=1)\n",
    "print(\"c.shape: \", c.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-03-03T04:56:13.192658Z",
     "start_time": "2022-03-03T04:56:13.181195Z"
    },
    "deletable": false
   },
   "outputs": [],
   "source": [
    "from torch.utils.data import Dataset\n",
    "class ECGDataset(Dataset):\n",
    "    def __init__(self, data_dict):\n",
    "        \"\"\"\n",
    "        TODO: init the Dataset instance.\n",
    "        \"\"\"\n",
    "        #  code\n",
    "  \n",
    "        self.dataset = data_dict\n",
    "    \n",
    "        # END\n",
    "\n",
    "    def __len__(self):\n",
    "        \"\"\"\n",
    "        TODO: Denotes the total number of samples\n",
    "        \"\"\"\n",
    "        # CODE \n",
    "        \n",
    "        return len(self.dataset['Y'])\n",
    "        #return len(self.y)\n",
    "    \n",
    "        # END\n",
    "\n",
    "    def __getitem__(self, i):\n",
    "        \"\"\"\n",
    "        TODO: Generates one sample of data as?\n",
    "            return the ((X, K_beat, K_rhythm, K_freq), Y) for the i-th data.\n",
    "            Do not return ((X, K_beat, K_rhythm, K_freq), Y)\n",
    "            Be careful which dimension you are indeXing.\n",
    "        \"\"\"\n",
    "        # CODE\n",
    "                   \n",
    "        return ((torch.tensor(self.dataset['X'][:, i, :], dtype = torch.float32),\n",
    "        torch.tensor(self.dataset['K_beat'][:, i, :], dtype = torch.float32),\n",
    "        torch.tensor(self.dataset['K_rhythm'][:, i, :], dtype = torch.float32),\n",
    "        torch.tensor(self.dataset['K_freq'][:, i, :], dtype = torch.float32)),\n",
    "        torch.tensor(self.dataset['Y'][i], dtype=torch.long))\n",
    "\n",
    "        # END\n",
    "### -----END CLASS DEF------ ###\n",
    "\n",
    "from torch.utils.data import DataLoader\n",
    "def load_data(dataset, batch_size=128):\n",
    "    \"\"\"\n",
    "    Return a DataLoader object with a Dataset object and and batch size.\n",
    "    Note that since the data has already been shuffled, we set shuffle=False\n",
    "    \"\"\"\n",
    "    \n",
    "    def my_collate(batch):\n",
    "\n",
    "        \"\"\"\n",
    "        param: batch: this is essentially [dataset[i] for i in [...]]\n",
    "        batch[i] should be ((Xi, Ki_beat, Ki_rhythm, Ki_freq), Yi)\n",
    "\n",
    "        TODO: write a collate function such that it outputs ((X, K_beat, K_rhythm, K_freq), Y)\n",
    "            each output variable is a batched version of what's in the input *batch*, essentially \n",
    "            [dataset[i] for i in [...]]\n",
    "            each output variable should be either float tensor, eXcept Y is long tensor. \n",
    "            If applicable, channel dim precedes batch dim\n",
    "            e.g. the shape of each Xi is (# channels, n). In the output, X should be of shape (# channels, batch_size, n)\n",
    "        \"\"\"\n",
    "        # CODE\n",
    "\n",
    "        # collect X1, X2, ... , X(LEN(BATCH), each w size = (# channels, n) = (4, 3000) into X.shape:(4, LEN(BATCH), 3000)\n",
    "\n",
    "        # batch is a (128 X 2) list, \n",
    "        # each rows is a two item tuple like (Xi, Ki_beat, Ki_rhythm, Ki_freq), Yi)\n",
    "#         \n",
    "        \n",
    "#         (list_X, list_K_beat, list_K_rhythm, list_K_freq), list_Y = zip(*batch)\n",
    "        \n",
    "#         (list_X, list_K_beat, list_K_rhythm, list_K_freq), list_Y = ([], [], [], []), []\n",
    "        # create zero tensors\n",
    "        X = torch.zeros(4, 0, 3000)\n",
    "        K_beat = torch.zeros(4, 0, 3000)\n",
    "        K_rhythm = torch.zeros(4, 0, 60)\n",
    "        K_freq = torch.zeros(4, 0, 1)\n",
    "        Y = torch.zeros(0)\n",
    "        \n",
    "        for i, element in enumerate(batch):\n",
    "            X = torch.cat((X, torch.tensor(element[0][0]).unsqueeze(dim=1)), dim = 1)\n",
    "            K_beat = torch.cat((K_beat, torch.tensor(element[0][1]).unsqueeze(dim=1)), dim = 1)\n",
    "            K_rhythm = torch.cat((K_rhythm, torch.tensor(element[0][2]).unsqueeze(dim=1)), dim = 1)\n",
    "            K_freq = torch.cat((K_freq, torch.tensor(element[0][3]).unsqueeze(dim=1)), dim = 1)\n",
    "\n",
    "            Y = torch.cat((Y, torch.tensor(element[1]).unsqueeze(dim=0)), dim=0)\n",
    "\n",
    "        # END\n",
    "        \n",
    "        return (X.float(), K_beat.float(), K_rhythm.float(), K_freq.float()), Y\n",
    "    \n",
    "        # my_collate takes batch = [dataset[i] for i in [...]]\n",
    "        # and returns batch[i], batch[1], batch[2], ... batch[batch_size?],  \n",
    "        # batch[i] = ((Xi, Ki_beat, Ki_rhythm, Ki_freq), Yi)\n",
    "        \n",
    "    # load_data takes dataset oject and batch size and returns a dataloader object.\n",
    "    return torch.utils.data.DataLoader(dataset, batch_size=batch_size, shuffle=False, collate_fn=my_collate)\n",
    "\n",
    "\n",
    "train_loader = load_data(ECGDataset(train_dict))\n",
    "test_loader = load_data(ECGDataset(test_dict))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why is there no test on test_loader. \n",
    "Why can I pass the test provided when test_loader cannot do what it is meant to do? next(iter(test_loader)) returns an error. \n",
    "\n",
    "Use this to run one iteration and get one batch.  \n",
    "`z = next(iter(test_loader)) `"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(np.asarray(list(zip(next(iter(test_loader))))).shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "next(iter(test_loader))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert iter(train_loader).next()[0][0].shape == (4, 128, 3000)\n",
    "assert iter(train_loader).next()[0][1].shape == (4, 128, 3000)\n",
    "assert iter(train_loader).next()[0][2].shape == (4, 128, 60)\n",
    "assert iter(train_loader).next()[1].shape == (128,)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-03-03T04:56:13.198777Z",
     "start_time": "2022-03-03T04:56:13.194564Z"
    },
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "7571ee37c076332323631e6347a23f32",
     "grade": false,
     "grade_id": "cell-1c2523e60e5ec87b",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "'''\n",
    "AUTOGRADER CELL. DO NOT MODIFY THIS.\n",
    "'''\n",
    "\n",
    "assert len(train_loader.dataset) == 1696, \"Length of training data incorrect.\"\n",
    "assert len(train_loader) == 14, \"Length of the training dataloader incorrect - maybe check batch_size\"\n",
    "assert [x.shape for x in train_loader.dataset[0][0]] == [(4,3000), (4,3000), (4,60), (4,1)], \"Shapes of the data don't match. Check __getitem__ implementation\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-03-03T04:56:13.203792Z",
     "start_time": "2022-03-03T04:56:13.200513Z"
    },
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "303c3eedeea50a80bbdc3c842a264202",
     "grade": true,
     "grade_id": "cell-b26f1973e85606c6",
     "locked": true,
     "points": 10,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "0e87a011f3ecfa5ebe1d203a491ab4b4",
     "grade": false,
     "grade_id": "cell-51b2320e646b960e",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "## 2 Model Defintions [75 points]\n",
    "\n",
    "Now, let us implement a model that involves RNN, CNN and attention mechanism. More specifically, we will implement [MINA: Multilevel Knowledge-Guided Attention for Modeling Electrocardiography Signals](https://www.ijcai.org/Proceedings/2019/0816.pdf).\n",
    "\n",
    "### 2.1 Knowledge-guided attention [15 points]\n",
    "Knowledge-guided attention is an attention mechanism that introduces prior knowledge (such as features proposed by human experts) in the features used by the attention mechanism. We will first define the general KnowledgeAttn module, and use it at different levels later.\n",
    "\n",
    "There are three steps:\n",
    "* 1\\. concatenate the input ($X$) and knowledge ($K$).\n",
    "* 2\\. pass it through a linear layer, a tanh, another linear layer, and softmax: $attn = softmax(V^\\top \\tanh(W^\\top \\begin{bmatrix}X\\\\K\\end{bmatrix}))$\n",
    "* 3\\. use attention values to sum $X$: $output = \\sum_{i=1}^D attn_i x_i$ where $attn_i$ is a scalar and $x_i$ is a vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-03-03T04:56:13.214241Z",
     "start_time": "2022-03-03T04:56:13.205461Z"
    },
    "deletable": false
   },
   "outputs": [],
   "source": [
    "import torch.nn.functional as F\n",
    "\n",
    "class KnowledgeAttn(nn.Module):\n",
    "    def __init__(self, input_features, attn_dim):\n",
    "        \"\"\"\n",
    "        This is the general knowledge-guided attention module.\n",
    "        \n",
    "        It will:\n",
    "        1. transform the input and knowledge with 2 linear layers\n",
    "        2. compute attention\n",
    "        3. aggregate.\n",
    "        \n",
    "        :param input_features: the number of features for each\n",
    "        :param attn_dim: the number of hidden nodes in the attention mechanism\n",
    "        \n",
    "        TODO:\n",
    "            define the following 2 linear layers WITHOUT bias (with the names provided)\n",
    "                att_W: a Linear layer of shape (input_features + n_knowledge, attn_dim)\n",
    "                att_v: a Linear layer of shape (attn_dim, 1)\n",
    "            init the weights using self.init() (already given)\n",
    "        \"\"\"\n",
    "        \n",
    "        super(KnowledgeAttn, self).__init__()\n",
    "        \n",
    "        self.input_features = input_features\n",
    "        self.attn_dim = attn_dim\n",
    "        self.n_knowledge = 1\n",
    "\n",
    "        # CODE\n",
    "        \n",
    "        # self.att_W = nn.Linear(input_features + n_knowledge, attn_dim)\n",
    "        # self.att_v = nn.Linear(attn_dim, 1)\n",
    "        \n",
    "        # n_knowledge not defined\n",
    "        \n",
    "        self.att_W = nn.Linear(self.input_features + self.n_knowledge, self.attn_dim, bias = False)\n",
    "        self.att_v = nn.Linear(self.attn_dim, 1, bias = False)\n",
    "        \n",
    "        # END\n",
    "\n",
    "        self.init() # init the weights\n",
    "\n",
    "    def init(self):\n",
    "        \n",
    "        nn.init.normal_(self.att_W.weight)\n",
    "        nn.init.normal_(self.att_v.weight)\n",
    "\n",
    "    @classmethod\n",
    "    def attention_sum(cls, x, attn):\n",
    "        \"\"\"\n",
    "\n",
    "        :param x: of shape (-1, D, nfeatures)\n",
    "        :param attn: of shape (-1, D, 1)\n",
    "        \n",
    "        TODO: return the weighted sum of x along the middle axis with weights even in attn. \n",
    "        output should be (-1, nfeatures)\n",
    "        \"\"\"\n",
    "        \n",
    "        # CODE\n",
    "        \n",
    "        # attn[1], i = 1, 2, ... D is a scalar, \n",
    "        # x[i], i = 1, 2, ..., D is a vector\n",
    "        # x[-1, 1, 1], vector multiplied elementwise by attn[-1, 1, 1], scalar\n",
    "        # x[-1, 2, 1], vector multiplied elementwise by attn[-1, 2, 1], scalar, \n",
    "        # plane is repeated allong dim=2 by broadcasting\n",
    "        ...\n",
    "        # in the equation above is attn[i]x[i] elementwise or dot product??\n",
    "        # x[-1, D, 1], \" \" \" \"   attn[-1, D, 1]\n",
    "        \n",
    "        \n",
    "        # output should be (-1, nfeatures)\n",
    "\n",
    "        return torch.sum(attn * x, dim = 1)\n",
    "        \n",
    "        # END\n",
    "\n",
    "\n",
    "    def forward(self, x, k):\n",
    "        \"\"\"\n",
    "        :param x: shape of (-1, D, input_features)\n",
    "        :param k: shape of (-1, D, 1)\n",
    "        :return:\n",
    "            out: shape of (-1, input_features), the aggregated x\n",
    "            attn: shape of (-1, D, 1)\n",
    "        TODO:\n",
    "            concatenate the input x and knowledge k together (on the last dimension)\n",
    "            pass the concatenated output through the learnable Linear transforms\n",
    "                first att_W, then tanh, then att_v\n",
    "                the output shape should be (-1, D, 1)\n",
    "            to get attention values, apply softmax on the output of linear layer\n",
    "                You could use F.softmax(). Be careful which dimension you apply softmax over\n",
    "            aggregate x using the attention values via self.attention_sum, and return\n",
    "        \"\"\"\n",
    "        # CODE\n",
    "\n",
    "        # concatenate the input x and knowledge k together (on the last dimension)\n",
    "\n",
    "        xk = torch.cat((x, k), dim = -1) \n",
    "                    \n",
    "        # pass the concatenated output through the learnable Linear transforms\n",
    "        \n",
    "        xk = self.att_W(xk)\n",
    "        # xk = self.tanh(xK) # 'KnowledgeAttn' object has no attribute 'tanh'\n",
    "        xk = torch.tanh(xk)\n",
    "        xk = self.att_v(xk) # out_features = 1  # xk.shape = (-1, D, 1) ??\n",
    "\n",
    "        # apply softmax on the output of linear layer to get attention values,\n",
    "        # You could use F.softmax(). Be careful which dimension you apply softmax over\n",
    "        \n",
    "        # attn: shape of (-1, D, 1)\n",
    "\n",
    "        attn =  F.softmax(xk, dim = 1)\n",
    "        \n",
    "        # aggregate x using the attention values via self.attention_sum and return.\n",
    "        # Q: Aggregate x or xk?\n",
    "        # A: x because the concatenation and transformations are only to get the attn values\n",
    "        # then attn values are simply applied to x\n",
    "        # out: shape of (-1, input_features), the aggregated x\n",
    "        \n",
    "        out = self.attention_sum(x, attn)            \n",
    "                  \n",
    "        # END\n",
    "        \n",
    "        return out, attn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-03-03T04:56:13.230131Z",
     "start_time": "2022-03-03T04:56:13.215676Z"
    },
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "cc0e3a1b0e748360df7c8586385da8a6",
     "grade": false,
     "grade_id": "cell-2d2af19c9cb95b68",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "'''\n",
    "AUTOGRADER CELL. DO NOT MODIFY THIS.\n",
    "'''\n",
    "\n",
    "def float_tensor_equal(a, b, eps=1e-3):\n",
    "    return torch.norm(a-b).abs().max().tolist() < eps\n",
    "\n",
    "def testKnowledgeAttn():\n",
    "    m = KnowledgeAttn(2, 2)\n",
    "    m.att_W.weight.data = torch.tensor([[0.3298,  0.7045, -0.1067],\n",
    "                                        [0.9656,  0.3090,  1.2627]], requires_grad=True)\n",
    "    m.att_v.weight.data = torch.tensor([[-0.2368,  0.5824]], requires_grad=True)\n",
    "\n",
    "    x = torch.tensor([[[-0.6898, -0.9098], [0.0230,  0.2879], [-0.2534, -0.3190]],\n",
    "                      [[ 0.5412, -0.3434], [0.0289, -0.2837], [-0.4120, -0.7858]]])\n",
    "    k = torch.tensor([[ 0.5469,  0.3948, -1.1430], [0.7815, -1.4787, -0.2929]]).unsqueeze(2)\n",
    "    out, attn = m(x, k)\n",
    "\n",
    "    tout = torch.tensor([[-0.2817, -0.2531], [0.2144, -0.4387]])\n",
    "    tattn = torch.tensor([[[0.3482], [0.4475], [0.2043]],\n",
    "                          [[0.5696], [0.1894], [0.2410]]])\n",
    "    assert float_tensor_equal(attn, tattn), \"The attention values are wrong\"\n",
    "    assert float_tensor_equal(out, tout), \"output of the attention module is wrong\"\n",
    "    \n",
    "testKnowledgeAttn()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-03-03T04:56:13.241157Z",
     "start_time": "2022-03-03T04:56:13.231726Z"
    },
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "1fd49ec2db01771c2b4e844647ebd828",
     "grade": true,
     "grade_id": "cell-ef150d8f011a0956",
     "locked": true,
     "points": 15,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "ed2282e718fd5e2c892f13807b7655fe",
     "grade": false,
     "grade_id": "cell-8ed326aa3362e5ed",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "### 2.2 MINA [60 points]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "722cb65c4fa05501621791b80d6ec980",
     "grade": false,
     "grade_id": "cell-048f0936985e30d0",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "We will now use the knowledge-guided attention mechanism to construct MINA. The overall structure is show below. From \"Input\" to \"Sliding Window Segmentation\" has already been done in the data preprocessing part, and in this section we will need to define things above \"Segment\"\n",
    "![MINAstructure](img/MINA_structure.png)\n",
    "\n",
    "\n",
    "Here, CNN (`BeatNet`) is used to capture beat information, Bi-LSTM (`RhythmNet`) is used to capture rhythm level information, and the from $c^{(i)}$ to $p$ is aggregating frequency levle infomration (`FreqNet`). Note that although the input has 4 channels, we actually need to handle each channel separately because they have different meanings after we did the FIR. Thus, we will need 4 `BeatNet`s, 4 `RhythmNet`s, and 1 `FreqNet`. \n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will need:\n",
    "- 4 CNN (BeatNet) n\n",
    "- 4 Bi-LSTM (RhythmNet)\n",
    "- one `FreqNet`, ie from  𝑐(𝑖) to  𝑝 above. \n",
    "\n",
    "`BeatNet` and `RhythmNet` act on each of four channels separately. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "7b0249a25b2e65554574f3ccf4385870",
     "grade": false,
     "grade_id": "cell-0d5314bb2c2c5ec6",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "MINA has three different knowledge guided attention mechanisms:\n",
    " - Beat Level $K_{beat}$: extract beat knowledge which is represented by the first-order difference and a convolutional operation $Conv_\\alpha$ for each segment\n",
    " - Rhythm Level $K_{rhythn}$: extract rhythm features represented by the standard deviation on each segment\n",
    " - Frequency Level $K_{freq}$: frequency features are represented by the power spectral density (PSD), which is a popular measure of energy in signal processing.\n",
    "\n",
    "#### 2.2.1 BeatNet [20 points]\n",
    "For BeatNet, the attention $\\alpha$ is computed by the following:\n",
    "    $$\\alpha = softmax(V_\\alpha^\\top \\tanh(W_\\alpha^\\top \\begin{bmatrix} \\mathbf{L}\\\\\\mathbf{K}_{beat} \\end{bmatrix}))$$\n",
    "Here, $L$ is output by the convolutional layers, and $K_{beat}$ is the computed beat level knowledge features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Attention for the CNN step/ beat level/local information:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-03-03T04:56:13.254433Z",
     "start_time": "2022-03-03T04:56:13.245270Z"
    },
    "deletable": false
   },
   "outputs": [],
   "source": [
    "class BeatNet(nn.Module):\n",
    "\n",
    "    def __init__(self, n=3000, T=50,\n",
    "                 conv_out_channels=64):\n",
    "        \"\"\"\n",
    "        :param n: size of each 10-second-data\n",
    "        :param T: size of each smaller segment used to capture local information in the CNN stage\n",
    "        :param conv_out_channels: also called number of filters/kernels\n",
    "        \n",
    "        TODO: We will define a network that does two things. Specifically:\n",
    "            1. use one 1-D convolutional layer to capture local informatoin, on x and k_beat, \n",
    "            the computed beat level knowledge features, (see forward())\n",
    "                conv: \n",
    "                 -The kernel size should be set to 32\n",
    "                 -the number of filters should be set to *conv_out_channels*. \n",
    "                 -Stride should be *conv_stride*\n",
    "                conv_k: same as conv, except that it has only 1 filter instead of *conv_out_channels*\n",
    "            2. an attention mechanism to aggregate the convolution outputs. Specifically:\n",
    "                attn: KnowledgeAttn with \n",
    "                -input_features equaling conv_out_channels, \n",
    "                -attn_dim = att_cnn_dim\n",
    "        \"\"\"\n",
    "        super(BeatNet, self).__init__() # multiple inheritance\n",
    "        self.n, self.M, self.T = n, int(n/T), T\n",
    "        self.conv_out_channels = conv_out_channels\n",
    "        self.conv_kernel_size = 32\n",
    "        self.conv_stride = 2\n",
    "        \n",
    "        # Define conv and conv_k, the two Conv1d modules\n",
    "        \n",
    "        # CODE\n",
    "        \n",
    "        self.conv = nn.Conv1d(in_channels = 1, \n",
    "                              out_channels = self.conv_out_channels, \n",
    "                              kernel_size = 32, \n",
    "                              stride = self.conv_stride\n",
    "                             )\n",
    "        \n",
    "        self.conv_k = nn.Conv1d(in_channels = 1, \n",
    "                              out_channels = 1, \n",
    "                              kernel_size = 32, \n",
    "                              stride = self.conv_stride\n",
    "                             )\n",
    "        \n",
    "        # END\n",
    "\n",
    "        self.att_cnn_dim = 8\n",
    "        \n",
    "        #Define attn, the KnowledgeAttn module\n",
    "        \n",
    "        # CODE\n",
    "        \n",
    "        self.attn = KnowledgeAttn(input_features = conv_out_channels, \n",
    "                                attn_dim = self.att_cnn_dim\n",
    "                                 )\n",
    "        self.rlu = nn.ReLU()\n",
    "        \n",
    "        # END\n",
    "\n",
    "    def forward(self, x, k_beat):\n",
    "        \"\"\"\n",
    "        :param x: shape (batch, n)\n",
    "        :param k_beat: shape (batch, n)\n",
    "        :return:\n",
    "            out: shape (batch, M, self.conv_out_channels)\n",
    "            alpha: shape (batch * M, N, 1) where N is a result of convolution\n",
    "        TODO:\n",
    "            [Given] reshape the data - convert x/k_beat of shape (batch, n) to (batch * M, 1, T), where n = MT\n",
    "                Use torch.Tensor.view() for all reshapes in this HW\n",
    "            apply convolution on x and k_beat\n",
    "                pass the reshaped x through self.conv, and then ReLU\n",
    "                pass the reshaped k_beat through self.conv_k, and then ReLU\n",
    "                        \n",
    "            (at this step, you might need to swap axix 1 & 2 to align the dimensions depending on how you defined the layers)\n",
    "            \n",
    "            pass the conv'd x and conv'd knowledge through self.attn to get the output (*out*) and attention (*alpha*)\n",
    "            [Given] reshape the output *out* to be of shape (batch, M, self.conv_out_channels)\n",
    "        \"\"\"\n",
    "        \n",
    "        #### Prior to given reshaping #### \n",
    "\n",
    "        # batch = 37, n = 408\n",
    "        \n",
    "        # convert x/k_beat of shape (batch, n) to (batch * M, 1, T), where n = MT\n",
    "        # M = 12\n",
    "        # T = 34\n",
    "        x = x.view(-1, self.T).unsqueeze(1)\n",
    "        k_beat = k_beat.view(-1, self.T).unsqueeze(1)\n",
    "                   \n",
    "        # CODE        \n",
    "\n",
    "        # x.shape:  torch.Size([444, 1, 34])\n",
    "\n",
    "        # k_beat.shape:  torch.Size([444, 1, 34])\n",
    "\n",
    "        x = self.conv(x)\n",
    "        x = self.rlu(x)\n",
    "        x = torch.permute(x, (0, 2, 1))\n",
    "        \n",
    "        k_beat = self.conv_k(k_beat)\n",
    "        k_beat = self.rlu(k_beat)\n",
    "        k_beat = torch.permute(k_beat, (0, 2, 1))\n",
    "\n",
    "        \n",
    "        # swap axis 1 & 2 so that self.attn runs.\n",
    "        # KnowledgeAttn uses torch.cat((x,k), dim = -1)\n",
    "        # this requries all but the last dimension to match\n",
    "        \n",
    "        out, alpha = self.attn(x, k_beat)\n",
    "        \n",
    "        # END\n",
    "        \n",
    "        out = out.view(-1, self.M, self.conv_out_channels)\n",
    "        \n",
    "        return out, alpha"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-03-03T04:56:13.290389Z",
     "start_time": "2022-03-03T04:56:13.256549Z"
    },
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "f58c92032e88c21c2dd5800c3a67c7fb",
     "grade": false,
     "grade_id": "cell-c5c9f279fa9197c0",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "'''\n",
    "AUTOGRADER CELL. DO NOT MODIFY THIS.\n",
    "'''\n",
    "_testm = BeatNet(12 * 34, 34, 56)\n",
    "assert isinstance(_testm.conv, torch.nn.Conv1d) and isinstance(_testm.conv_k, torch.nn.Conv1d), \"Should use nn.Conv1d\"\n",
    "assert _testm.conv.bias.shape == torch.Size([56]) and _testm.conv.weight.shape == torch.Size([56,1,32]), \"conv definition is incorrect\"\n",
    "assert _testm.conv_k.bias.shape == torch.Size([1]) and _testm.conv_k.weight.shape == torch.Size([1, 1, 32]), \"conv_k definition is incorrect\"\n",
    "assert isinstance(_testm.attn, KnowledgeAttn), \"Should use one KnowledgeAttn Module\"\n",
    "\n",
    "_out, _alpha =_testm(torch.randn(37, 12*34), torch.randn(37, 12*34))\n",
    "assert _alpha.shape == torch.Size([444,2,1]), \"The attention's dimension is incorrect\"\n",
    "assert _out.shape==torch.Size([37, 12,56]), \"The output's dimension is incorrect\"\n",
    "del _testm, _out, _alpha"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-03-03T04:56:13.301889Z",
     "start_time": "2022-03-03T04:56:13.291856Z"
    },
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "7f8e5a019e8424e5992b24e45c2f7bc1",
     "grade": true,
     "grade_id": "cell-bd0c0614df001bfc",
     "locked": true,
     "points": 20,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "d34a00d6630667a2e27cc1d1e3df2f60",
     "grade": false,
     "grade_id": "cell-143d78d2776ddbed",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "#### 2.2.2 RhythmNet [20 points]\n",
    "For Rhythm, the attention $\\beta$ is computed by the following:\n",
    "    $$\\beta = softmax(V_\\beta^\\top \\tanh(W_\\beta^\\top \\begin{bmatrix} \\mathbf{H}\\\\\\mathbf{K}_{rhythm} \\end{bmatrix}))$$\n",
    "Here, $\\mathbf{H}$ is output by the Bi-LSTMs, and $K_{rhythm}$ is the computed rhythm level knowledge features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TODO: We will define a network that does two things to handle rhythms. Specifically:\n",
    "    1. use a bi-directional LSTM to process the learned local representations from the CNN part        \n",
    "    2. an attention mechanism to aggregate the convolution outputs. Specifically:\n",
    "    3. output layers\n",
    "    \n",
    "    \n",
    "[torch.nn.LSTM()](https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html)  \n",
    "See:\n",
    "- parameters\n",
    "- inputs: input, (h_0, c_0)\n",
    "- output: output, (h_n, c_n)\n",
    "- examples:\n",
    "```\n",
    "    rnn = nn.LSTM(10, 20, 2)\n",
    "    input = torch.randn(5, 3, 10)\n",
    "    h0 = torch.randn(2, 3, 20)\n",
    "    c0 = torch.randn(2, 3, 20)\n",
    "    output, (hn, cn) = rnn(input, (h0, c0))\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### torch.nn.Linear()\n",
    "Applies linear trans to input, $O_{ut} = I_nW_{eights}^T+b$  \n",
    "Takes in_f, out_f = last dim of in and out matrices. All other dims must match.\n",
    "weights.shape = (out_f, in_f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-03-03T04:56:13.312189Z",
     "start_time": "2022-03-03T04:56:13.303881Z"
    },
    "deletable": false
   },
   "outputs": [],
   "source": [
    "class RhythmNet(nn.Module):\n",
    "    def __init__(self, n=3000, T=50, input_size=64, rhythm_out_size=8):\n",
    "        \"\"\"\n",
    "        :param n: size of each 10-second-data\n",
    "        :param T: size of each smaller segment used to capture local information in the CNN stage\n",
    "        :param input_size: This is the same as the # of filters/kernels in the CNN part.\n",
    "        :param rhythm_out_size: output size of this netowrk\n",
    "        \n",
    "        \"\"\"\n",
    "        \n",
    "        #input_size is the cnn_out_channels\n",
    "        super(RhythmNet, self).__init__()\n",
    "        self.n, self.M, self.T = n, int(n/T), T\n",
    "        self.input_size = input_size\n",
    "\n",
    "        self.rnn_hidden_size = 32\n",
    "        \n",
    "        ### define lstm: LSTM Input is of shape (batch size, M, input_size)\n",
    "       \n",
    "        # CODE  \n",
    "        \n",
    "        # lstm: bidirectional, 1 layer, batch_first, and hidden_size should be set to *rnn_hidden_size*\n",
    "        # size = features \n",
    "        \n",
    "        \n",
    "        self.lstm = nn.LSTM(input_size = self.input_size, \n",
    "                           hidden_size = self.rnn_hidden_size, \n",
    "                           bidirectional = True, \n",
    "                           batch_first = True\n",
    "                          )\n",
    "        # END        \n",
    "\n",
    "        ### Attention mechanism: define attn to be a KnowledgeAttn\n",
    "        \n",
    "        self.att_rnn_dim = 8\n",
    "        \n",
    "        # CODE  \n",
    "        \n",
    "        # attn: KnowledgeAttn with input_features equaling lstm output ???, and attn_dim=att_rnn_dim\n",
    "        \n",
    "        # the LSTM output size is the hidden_size argument, except in bidir is 2 times that.\n",
    "        # When you have a bidirectional LSTM, it concatenates the forward and reverse direction.\n",
    "        # So, your input size to attn will be double that of the hidden_size which in this case is rnn_hidden_size\n",
    "        \n",
    "        self.attn = KnowledgeAttn(input_features = 2 * self.rnn_hidden_size, attn_dim = self.att_rnn_dim)\n",
    "    \n",
    "        # END        \n",
    "\n",
    "        ### Define the Dropout and fully connecte layers (fc and do)\n",
    "        \n",
    "        self.out_size = rhythm_out_size\n",
    "        \n",
    "        # CODE\n",
    "\n",
    "        # fc: a Linear layer making the output of shape (..., self.out_size)\n",
    "        # weights.shape = (self.out_size, 2 * self.rnn_hidden_size)\n",
    "        self.fc = nn.Linear(in_features = 2 * self.rnn_hidden_size, out_features = self.out_size)\n",
    "        \n",
    "        # do: a Dropout layer with p=0.5, default        \n",
    "        self.do = nn.Dropout()\n",
    "        \n",
    "        self.rl = nn.ReLU()\n",
    "        \n",
    "        # END\n",
    "\n",
    "    def forward(self, x, k_rhythm):\n",
    "        \"\"\"\n",
    "        :param x: shape (batch, M, self.input_size)\n",
    "        :param k_rhythm: shape (batch, M)\n",
    "        :return:\n",
    "            out: shape (batch, self.out_size)\n",
    "            beta: shape (batch, M, 1)\n",
    "        \"\"\"\n",
    "\n",
    "        # CODE\n",
    "          \n",
    "        \n",
    "        # reshape the k_rhythm->(batch, M, 1) (HINT: use k_rhythm.unsqueeze())    \n",
    "        k_rhythm = k_rhythm.unsqueeze(dim = 2)\n",
    "        \n",
    "        # pass the RESHAPED x through lstm\n",
    "        output, (h_n, c_n) = self.lstm(x) \n",
    "        # output, (hn, cn) = self.lstm(input, (h0, c0)) \n",
    "        \n",
    "        # denote the final output as *out*, and the attention output as *beta* \n",
    "        # pass the lstm output and knowledge through attn \n",
    "        # assign the results to out, beta \n",
    "  \n",
    "        out, beta = self.attn(output, k_rhythm) # beta = out, pass the aggregated x values.\n",
    "        \n",
    "        # pass the output results, not the attn, through fully connected layer - ReLU - Dropout        \n",
    "\n",
    "        out = self.fc(out.expand(-1, 2 * self.rnn_hidden_size))    \n",
    "        out = self.rl(out)\n",
    "        out = self.do(out)\n",
    "\n",
    "        # END\n",
    "            \n",
    "        return out, beta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-03-03T04:56:13.332845Z",
     "start_time": "2022-03-03T04:56:13.314053Z"
    },
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "d6d2b2bb4c79ca1984b7b03d1b0a9eef",
     "grade": false,
     "grade_id": "cell-35c5e07324375dec",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "'''\n",
    "AUTOGRADER CELL. DO NOT MODIFY THIS.\n",
    "'''\n",
    "_B, _M, _T = 17, 23, 31\n",
    "_testm = RhythmNet(_M * _T, _T, 37)\n",
    "assert isinstance(_testm.lstm, torch.nn.LSTM), \"Should use nn.LSTM\"\n",
    "assert _testm.lstm.bidirectional, \"LSTM should be bidirectional\"\n",
    "assert isinstance(_testm.attn, KnowledgeAttn), \"Should use one KnowledgeAttn Module\"\n",
    "assert isinstance(_testm.fc, nn.Linear) and _testm.fc.weight.shape == torch.Size([8,64]), \"The fully connected is incorrect\"\n",
    "assert isinstance(_testm.do, nn.Dropout), \"Dropout layer is not defined correctly\"\n",
    "\n",
    "_out, _beta = _testm(torch.randn(_B, _M, 37), torch.randn(_B, _M))\n",
    "assert _beta.shape == torch.Size([_B,_M,1]), \"The attention's dimension is incorrect\"\n",
    "assert _out.shape==torch.Size([_B, 8]), \"The output's dimension is incorrect\"\n",
    "del _testm, _out, _beta,  _B, _M, _T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-03-03T04:56:13.346301Z",
     "start_time": "2022-03-03T04:56:13.334511Z"
    },
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "e0fb649e72671e1250559ce986af4763",
     "grade": true,
     "grade_id": "cell-8b673254a669937b",
     "locked": true,
     "points": 20,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "ce88267566036dea2817486941e2fb52",
     "grade": false,
     "grade_id": "cell-e9434c36630bceaa",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "#### 2.2.3 FreqNet [20 points]\n",
    "The attention $\\gamma$ is computed by the following:\n",
    "    $$\\gamma = softmax(V_\\gamma^\\top \\tanh(W_\\gamma^\\top \\begin{bmatrix} \\mathbf{Q}\\\\\\mathbf{K}_{freq} \\end{bmatrix}))$$\n",
    "Here, $\\mathbf{Q}$ is output of the RhythmNets, and $K_{freq}$ is the computed frequency level knowledge features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the main network that orchestrates the previously defined attention modules:  \n",
    "A. definition ie `init_`:\n",
    "1. define n_channels many BeatNet and RhythmNet modules. e\n",
    "2. define frequency (channel) level knowledge-guided attention module\n",
    "3. output layer: a Linear layer for 2 classes output\n",
    "\n",
    "Execution: ie `forward` method:\n",
    "Use the attention submodules to process data from each channel separately, and then pass the\n",
    "    output through an attention on frequency for the final output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**nn.ModuleList()**  \n",
    "Holds submodules in a list.  \n",
    "  \n",
    "ModuleList can be indexed like a regular Python list, but modules it contains are properly registered, and will be visible by all Module methods.  \n",
    "\n",
    "*Example:*  \n",
    "```\n",
    "class MyModule(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.linears = nn.ModuleList([nn.Linear(10, 10) for i in range(10)])\n",
    "\n",
    "    def forward(self, x):\n",
    "        # ModuleList can act as an iterable, or be indexed using ints\n",
    "        for i, l in enumerate(self.linears):\n",
    "            x = self.linears[i // 2](x) + l(x)\n",
    "        return x\n",
    "```\n",
    "\n",
    "**append()**\n",
    "Appends a given module of nn.Module type to the end of the list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-03-03T04:56:13.357720Z",
     "start_time": "2022-03-03T04:56:13.348099Z"
    },
    "deletable": false
   },
   "outputs": [],
   "source": [
    "from IPython.core.debugger import set_trace\n",
    "\n",
    "class FreqNet(nn.Module):\n",
    "    def __init__(self, n_channels=4, n=3000, T=50):\n",
    "        \"\"\"\n",
    "        This is the main network that orchestrates the previously defined attention modules:\n",
    "        \"\"\"\n",
    "        super(FreqNet, self).__init__()\n",
    "               \n",
    "        self.n, self.M, self.T = n, int(n / T), T\n",
    "        # :param n_channels: number of channels (F in the paper). Define #channels # of BeatNet & RhythmNet nets.\n",
    "        # :param n: size of each 10-second-data\n",
    "        # :param T: size of each smaller segment used to capture local information in the CNN stage  \n",
    "        \n",
    "        self.n_class = 2\n",
    "        self.n_channels = n_channels\n",
    "                \n",
    "        self.conv_out_channels = 64\n",
    "        self.rhythm_out_size = 8\n",
    "        self.beat_nets = nn.ModuleList()\n",
    "        self.rhythm_nets = nn.ModuleList()\n",
    "        \n",
    "        #use self.beat_nets.append() and self.rhythm_nets.append() to append 4 BeatNets/RhythmNets\n",
    "        \n",
    "        # CODE\n",
    "        \n",
    "        # 1. define n_channels many BeatNet and RhythmNet modules. (Hint: use nn.ModuleList)\n",
    "            # beat_nets: for each, pass parameter conv_out_channel into the init()\n",
    "            # rhythm_nets: for each, pass conv_out_channel as input_size,\n",
    "            #                    and self.rhythm_out_size as the output size\n",
    "\n",
    "        for i in range(self.n_channels):\n",
    "            self.beat_nets.append(BeatNet(n = self.n,\n",
    "                                          T = self.T,\n",
    "                                          conv_out_channels=self.conv_out_channels\n",
    "                                         )\n",
    "                                 )\n",
    "            \n",
    "            self.rhythm_nets.append(RhythmNet(n = self.n, \n",
    "                                              T = self.T, \n",
    "                                              input_size = self.conv_out_channels, \n",
    "                                              rhythm_out_size = self.rhythm_out_size\n",
    "                                             )\n",
    "                                   )        \n",
    "        # END \n",
    "\n",
    "\n",
    "        self.att_channel_dim = 2\n",
    "        \n",
    "        # 2. define frequency (channel) level knowledge-guided attention module\n",
    "          # attn: KnowledgeAttn with input_features equaling rhythm_out_size, and attn_dim=att_channel_dim\n",
    "        \n",
    "        # CODE\n",
    "        \n",
    "        self.attn = KnowledgeAttn(input_features = self.rhythm_out_size, \\\n",
    "                                  attn_dim = self.att_channel_dim\n",
    "                                 )\n",
    "        # END \n",
    "\n",
    "        ### Create the fully-connected output layer (fc)\n",
    "        \n",
    "        # 3. output layer: a Linear layer for 2 classes output\n",
    "\n",
    "        # CODE\n",
    "        \n",
    "        self.fc = nn.Linear(in_features = self.rhythm_out_size, \n",
    "                            out_features = self.n_class) ### ???\n",
    "       \n",
    "        # END \n",
    "\n",
    "\n",
    "        \"\"\"\n",
    "        :param x: shape (n_channels, batch, n)\n",
    "        :param k_beats: (n_channels, batch, n)\n",
    "        :param k_rhythms: (n_channels, batch, M)\n",
    "        :param k_freq: (n_channels, batch, 1)\n",
    "        \n",
    "        :return:\n",
    "            out: softmax output for each data point, shpae (batch, n_class)\n",
    "            gama: the attention value on channels\n",
    "            \n",
    "        TODO:\n",
    "            1. [Given] pass each channel of x through the corresponding beat_net, then rhythm_net.\n",
    "                We will discard the attention (alpha and beta) outputs for now\n",
    "                Using ModuleList for self.beat_nets/rhythm_nets is necessary for the gradient to propagate\n",
    "            2. [Given] stack the output from 1 together into a tensor of shape (batch, n_channels, rhythm_out_size)\n",
    "            3. pass result from 2 and k_freq through attention module, to get the aggregated result and *gama*\n",
    "                You might need to do use k_freq.permute() to tweak the shape of k_freq\n",
    "            4. pass aggregated result from 3 through the final fully connected layer.\n",
    "            5. Apply Softmax to normalize output to a probability distribution (over 2 classes)\n",
    "        \"\"\"\n",
    "    def forward(self, x, k_beats, k_rhythms, k_freq):       \n",
    "        \n",
    "        new_x = [None for _ in range(self.n_channels)]\n",
    "        \n",
    "        # 1. pass each channel of x through the corresponding beat_net, then rhythm_net.\n",
    "            # Do not collect the attention outputs\n",
    "            # We used ModuleList so that the gradient would propagate\n",
    "            \n",
    "        for i in range(self.n_channels):\n",
    "            tx, _ = self.beat_nets[i](x[i], k_beats[i])\n",
    "            new_x[i], _ = self.rhythm_nets[i](tx, k_rhythms[i])\n",
    "            \n",
    "            \n",
    "        # 2. stack the output into a tensor with shape (batch, n_channels, rhythm_out_size)\n",
    "        # output shape: [128,8]\n",
    "#         stack along dim = 1 to get [128,4,8]\n",
    "        x = torch.stack(new_x, 1)    \n",
    "\n",
    "        # CODE\n",
    "        \n",
    "        # 3. pass result from 2 and k_freq through attention module, to get the aggregated result and *gama*\n",
    "        # k_freq.shape:  torch.Size([4, 17, 1]) => 17, 4, 1\n",
    "        # out: shape of (-1, input_features)\n",
    "\n",
    "        out, gama = self.attn(x, torch.permute(k_freq, (1, 0, 2)))\n",
    "\n",
    "        # 4. pass aggregated result from 3 through the final fully connected layer.\n",
    "        out = self.fc(out)\n",
    "        \n",
    "        # 5. Apply Softmax to normalize output to a probability distribution (over 2 classes)\n",
    "        out = torch.softmax(out, dim = 1)\n",
    "        \n",
    "        # EMD\n",
    "        \n",
    "        return out, gama"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-03-03T04:56:13.521085Z",
     "start_time": "2022-03-03T04:56:13.359430Z"
    },
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "dc33b0b4b98c828b4f6b37c7a18719c3",
     "grade": false,
     "grade_id": "cell-732a60c51776cfe1",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "'''\n",
    "AUTOGRADER CELL. DO NOT MODIFY THIS.\n",
    "'''\n",
    "_B, _M, _T = 17, 59, 109\n",
    "_testm = FreqNet(n=_M * _T, T=_T)\n",
    "assert isinstance(_testm.attn, KnowledgeAttn), \"Should use one KnowledgeAttn Module\"\n",
    "assert isinstance(_testm.fc, nn.Linear) and _testm.fc.weight.shape == torch.Size([2,8]), \"The fully connected is incorrect\"\n",
    "assert isinstance(_testm.beat_nets, nn.ModuleList), \"beat_nets has to be a ModuleList\"\n",
    "\n",
    "_out, _gamma = _testm(torch.randn(4, _B, _M * _T), torch.randn(4, _B, _M * _T), torch.randn(4, _B, _M), torch.randn(4, _B, 1))\n",
    "assert _gamma.shape == torch.Size([_B, 4, 1]), \"The attention's dimension is incorrect\"\n",
    "assert _out.shape==torch.Size([_B, 2]), \"The output's dimension is incorrect\"\n",
    "del _testm, _out, _gamma,  _B, _M, _T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-03-03T04:56:13.602043Z",
     "start_time": "2022-03-03T04:56:13.522613Z"
    },
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "8e0c35f8494e9f5fcae91481c68ea679",
     "grade": true,
     "grade_id": "cell-82b587bd3f11fae0",
     "locked": true,
     "points": 20,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "e96ab553a56bc7073495c7d5b8616354",
     "grade": false,
     "grade_id": "cell-21afb56d006509f1",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "## 3 Training and Evaluation [15 points]\n",
    "In this part we will define the training procedures, train the model, and evaluate the model on the test set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**train_model parameters:**\n",
    "- model: The instance of FreqNet that we are training\n",
    "- train_dataloader: the DataLoader of the training data\n",
    "- n_epoch: number of epochs to train\n",
    "- lr: learning rate\n",
    "- device: cpu or gpu/cuda\n",
    "\n",
    "**return/output:**\n",
    "- _model): trained model\n",
    "- _loss_history_: recorded training loss history - should be just a list of float"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**train_model tasks:**  \n",
    "I. Specify the optimizer (*optimizer*) to be optim.Adam  \n",
    "II. Specify the loss function (*loss_func*) to be CrossEntropyLoss  \n",
    "III. Within the loop, do the normal training procedures:  \n",
    "- A. pass the input through the model  \n",
    "- B. pass the output through loss_func to compute the loss  \n",
    "- C. zero out currently accumulated gradient, use loss.basckward to backprop the gradients, then call optimizer.step  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**eval_model tasks:**  \n",
    "  \n",
    "**returns:**  \n",
    "  \n",
    ">**pred_all:** prediction of model on the dataloder.  \n",
    "    >>_Should be an 2D numpy float array where the second dimension has length 2._\n",
    "    \n",
    ">**Y_test:** truth labels. Should be an numpy array of ints  \n",
    "      \n",
    "**tasks:**  \n",
    "1. evaluate the model using on the data in the dataloder.  \n",
    "2. Add all the prediction and truth to the corresponding list  \n",
    "3. Convert pred_all and Y_test to numpy arrays (of shape (n_data_points, 2))  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[ADAM](https://pytorch.org/docs/stable/generated/torch.optim.Adam.html)  \n",
    "params (iterable) – iterable of parameters to optimize or dicts defining parameter groups\n",
    "```\n",
    "torch.optim.Adam(\n",
    "                params, \n",
    "                lr=0.001, \n",
    "                betas=(0.9, 0.999), \n",
    "                eps=1e-08, \n",
    "                weight_decay=0, \n",
    "                amsgrad=False, *, \n",
    "                foreach=None, \n",
    "                maximize=False, \n",
    "                capturable=False, \n",
    "                differentiable=False, \n",
    "                fused=None\n",
    "                )\n",
    "```\n",
    "\n",
    "[CrossEntropyLoss](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html)  \n",
    "```\n",
    "torch.nn.CrossEntropyLoss(\n",
    "                        weight=None, \n",
    "                        ignore_index=- 100, \n",
    "                        reduction='mean', \n",
    "                        label_smoothing=0.0\n",
    "                        )\n",
    "```\n",
    "loss between input logits and target  \n",
    "**input:** unnormalized logits for each class.  \n",
    "input has to be a Tensor of size (C), for unbached input (minibatch, C) or (minibatch, c, d1, d2, ... dk) for d = dim.  \n",
    "**target:**  \n",
    "Class indices in the range (0, C), C = number of classes (_preferred_)  \n",
    "or    \n",
    "Probabilities for each class  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-03-03T04:56:13.612528Z",
     "start_time": "2022-03-03T04:56:13.603547Z"
    },
    "deletable": false
   },
   "outputs": [],
   "source": [
    "def train_model(model, train_dataloader, n_epoch=5, lr=0.003, device=None):\n",
    "    import torch.optim as optim\n",
    "\n",
    "    device = device or torch.device('cpu')\n",
    "    model.train()\n",
    "\n",
    "    loss_history = []\n",
    "\n",
    "    # CODE\n",
    "    \n",
    "    # I. Specify the optimizer (*optimizer*) to be optim.Adam  \n",
    "    \n",
    "    optimizer = optim.Adam(model.parameters(), lr=lr) \n",
    "    # II. Specify the loss function (loss_func) to be CrossEntropyLoss\n",
    "    loss_func = nn.CrossEntropyLoss()\n",
    "    \n",
    "    # END\n",
    "\n",
    "    for epoch in range(n_epoch):\n",
    "        curr_epoch_loss = []\n",
    "        for (X, K_beat, K_rhythm, K_freq), Y in train_dataloader:\n",
    "#             print(\"########## X.shape: ############\", X.shape)\n",
    "            # CODE\n",
    "            \n",
    "            # III. Within the loop, do the normal training procedures:                \n",
    "            pred = model(X, K_beat, K_rhythm, K_freq)[0] # Y = model(X)  ## ???  # A. pass the input through the model                      \n",
    "            loss = loss_func(pred, Y.long())# B. pass the output through loss_func to compute the loss\n",
    "\n",
    "            # C.\n",
    "            optimizer.zero_grad() # i. zero out currently accumulated gradient, \n",
    "            loss.backward() # ii. use loss.basckward to backprop the gradient\n",
    "            optimizer.step() # iii. call optimizer.step\n",
    "    \n",
    "            # END\n",
    "            \n",
    "            curr_epoch_loss.append(loss.cpu().data.numpy())\n",
    "        print(f\"epoch{epoch}: curr_epoch_loss={np.mean(curr_epoch_loss)}\")\n",
    "        loss_history += curr_epoch_loss\n",
    "    return model, loss_history\n",
    "\n",
    "def eval_model(model, dataloader, device=None):\n",
    "\n",
    "    device = device or torch.device('cpu')\n",
    "    model.eval()\n",
    "    pred_all = []\n",
    "    Y_test = []\n",
    "    for (X, K_beat, K_rhythm, K_freq), Y in dataloader:\n",
    "        \n",
    "        # CODE\n",
    "        \n",
    "        # 1. evaluate the model using on the data in the dataloder.  \n",
    "        # Should be an 2D numpy float array where the second dimension has length 2.\n",
    "        # 2. Add all the prediction and truth to the corresponding list        \n",
    "        # 3. Convert pred_all and Y_test to numpy arrays (of shape (n_data_points, 2))        \n",
    "        pred_all.append(model(X, K_beat, K_rhythm, K_freq)[0].detach().numpy().astype(float))\n",
    "        Y_test.append(Y.numpy())\n",
    "      \n",
    "        # END \n",
    "        \n",
    "    pred_all = np.concatenate(pred_all, axis=0)\n",
    "    Y_test = np.concatenate(Y_test, axis=0)\n",
    "\n",
    "    return pred_all, Y_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-03-03T04:56:13.616456Z",
     "start_time": "2022-03-03T04:56:13.614202Z"
    },
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "11924f8ec3adb10d31664434b49ccd50",
     "grade": true,
     "grade_id": "cell-85a8b3001bf2666b",
     "locked": true,
     "points": 0,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-03-03T04:57:02.605035Z",
     "start_time": "2022-03-03T04:56:13.618219Z"
    },
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "eb80c26eeb402d555cd006495aad3966",
     "grade": false,
     "grade_id": "cell-b518dc80db078e45",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "device = torch.device('cpu')\n",
    "n_epoch = 4\n",
    "lr = 0.003\n",
    "n_channel = 4\n",
    "n_dim=3000\n",
    "T=50\n",
    "\n",
    "model = FreqNet(n_channel, n_dim, T)\n",
    "model = model.to(device)\n",
    "\n",
    "model, loss_history = train_model(model, train_loader, n_epoch=n_epoch, lr=lr, device=device)\n",
    "pred, truth = eval_model(model, test_loader, device=device)\n",
    "#pd.to_pickle((pred, truth), \"./deliverable.pkl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred, truth = eval_model(model, test_loader, device=device)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred.shape, truth.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%debug"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`f1_score(y_true, y_pred)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-03-03T04:57:02.615221Z",
     "start_time": "2022-03-03T04:57:02.607736Z"
    },
    "deletable": false
   },
   "outputs": [],
   "source": [
    "def evaluate_predictions(truth, pred):\n",
    "    \"\"\"\n",
    "    TODO: Evaluate the performance of the predictoin via AUROC, and F1 score\n",
    "    each prediction in pred is a vector representing [p_0, p_1].\n",
    "    When defining the scores we are interesed in detecting class 1 only, ie 0, 1\n",
    "    (Hint: use roc_auc_score and f1_score from sklearn.metrics, be sure to read their documentation)\n",
    "    return: auroc, f1\n",
    "    \"\"\"\n",
    "    from sklearn.metrics import roc_auc_score, f1_score\n",
    "\n",
    "    auroc = roc_auc_score(truth, pred[:, 1])\n",
    "    pred = np.argmax(pred, axis=1)\n",
    "    f1=f1_score(truth, pred)\n",
    "\n",
    "    return auroc, f1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-03-03T04:57:04.569925Z",
     "start_time": "2022-03-03T04:57:02.617925Z"
    },
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "d706fd2e815507fc52ffabb92b808f82",
     "grade": false,
     "grade_id": "cell-e32529de46905343",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "'''\n",
    "AUTOGRADER CELL. DO NOT MODIFY THIS.\n",
    "'''\n",
    "pred, truth = eval_model(model, test_loader, device=device)\n",
    "auroc, f1 = evaluate_predictions(truth, pred)\n",
    "print(f\"AUROC={auroc} and F1={f1}\")\n",
    "\n",
    "assert auroc > 0.8 and f1 > 0.7, \"Performance is too low {}. Something's probably off.\".format((auroc, f1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-03-03T04:57:06.058218Z",
     "start_time": "2022-03-03T04:57:04.571364Z"
    },
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "1bcf4542187b41e3efb45ce989d5da40",
     "grade": true,
     "grade_id": "cell-5a6f298a1e4cf932",
     "locked": true,
     "points": 15,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "illinois_payload": {
   "b64z": "",
   "nb_path": "release/HW4_MINA/HW4_MINA.ipynb"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}