lectures/lecture-2/lecture-2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Lecture 2:  Convolutional neural networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Previous lecture: Basic concepts\n",
    "- **General discussion**: how deep learning is different from classical machine learning\n",
    "- **Supervised/unsupervised learning**: abstract formulation\n",
    "- **Fully connected neural networks**: What it is and why depth matters\n",
    "- **The concept of backpropagation and 'cheap gradients'**: why it is important to compute the gradient in a fast way (and how)\n",
    "- **Convolutional neural networks**: Brief definition of the CNN (to be followed tomorrow)\n",
    "- **Popular deep learning libraries**: Tensorflow, Pytorch and Jax."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Todays lecture\n",
    "\n",
    "Convolutional neural networks in more details:\n",
    "\n",
    "- Motivation of using convolutions (and connection to classical image processing)\n",
    "- Basic building blocks of a CNN \n",
    "- Overview of the main architectures (LeNet, AlexNet, VGG, ResNet, Inception, EfficientNet, MobileNet)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Why convolutions for images?\n",
    "\n",
    "- Images exhibit locality properties.\n",
    "- Many classical image transformations can be written down as convolutions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Some transforms\n",
    "\n",
    "- How to **sharpen** the image? \n",
    "- Edge detect? \n",
    "- Strong edge detect?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Pretrained filters of AlexNet\n",
    "\n",
    "The first layer of the AlexNet model gives the following filters.\n",
    "\n",
    "They are nice and smooth, which indicates the network has trained!\n",
    "\n",
    "Note, that there are other visualization tools for the inner parameters of artificial neural networks!\n",
    "\n",
    "<img src=\"filt1.jpeg\" width=\"40%\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Cross correlation\n",
    "\n",
    "Strictly speaking, it is not a convolution, but **cross-correlation**.\n",
    "\n",
    "<img src=\"correlation.svg\" width=\"80%\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Padding and strides\n",
    "\n",
    "The convolutions have problems with the boundary; If we do it many times, it will reduce the width and height of the image.\n",
    "\n",
    "A typical solution is too add pixels around the border. \n",
    "\n",
    "<img src=\"conv-reuse.svg\" width=\"80%\">\n",
    "\n",
    "We can also shift the window not by 1 row/column, but by larger **strides**. \n",
    "This will downsample the object."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Strides\n",
    "\n",
    "<img src=\"conv-stride.svg\" width=\"80%\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## LeNet\n",
    "\n",
    "The first convolutional neural network architecture architecture was LeNet (tested on MNIST dataset).\n",
    "\n",
    "\n",
    "<img src=\"lenet.svg\" width=\"80%\">\n",
    "\n",
    "What is **average pooling** ? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Average pooling\n",
    "\n",
    "- Pooling is a common operation in CNNs used to downsample feature maps\n",
    "- Average pooling takes the average value of each sub-region of the input feature map\n",
    "- It reduces the size of the feature map while introducing a form of spatial invariance\n",
    "- Average pooling helps to reduce overfitting and improve generalization\n",
    "- However, it may lose information about the precise location of features\n",
    "- Alternatives to average pooling include **max pooling**, **adaptive pooling**\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([1, 64, 5, 7])\n"
     ]
    }
   ],
   "source": [
    "#Demo of adaptive & average poolingx\n",
    "m = nn.AdaptiveAvgPool2d((5, 7))\n",
    "input = torch.randn(1, 64, 8, 9)\n",
    "output = m(input)\n",
    "print(output.shape)\n",
    "# target output size of 7x7 (square)\n",
    "m = nn.AdaptiveAvgPool2d(7)\n",
    "input = torch.randn(1, 64, 10, 9)\n",
    "output = m(input)\n",
    "# target output size of 10x7\n",
    "m = nn.AdaptiveAvgPool2d((None, 7))\n",
    "input = torch.randn(1, 64, 10, 9)\n",
    "output = m(input)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Performance of LeNet\n",
    "\n",
    "\n",
    "LeNet has been tested on **MNIST** dataset (handwritten digits, 60000 samples).\n",
    "\n",
    "This is an easy task.\n",
    "\n",
    "Later, more complicated datasets have been introduced (CIFAR-10, CIFAR-100, Pascal VOC).\n",
    "\n",
    "But they were still small and low-resolution, deep learning at that moment did not seem very promising for them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## ImageNet dataset\n",
    "\n",
    "In 2006, Fei-Fei Li proposed the creation of an ImageNet dataset.\n",
    "\n",
    "'While most of the people pay attention to models, lets pay attention to data'.\n",
    "\n",
    "In July 2008, ImageNet had zero images. By December, it had categorized three million images across 6,000+ synsets. In April 2010, there were more than 11 million images in 15,000+ synsets. Such results would have been inconceivable for a handful of researchers. They were made possible through crowdsourcing on Amazon’s Mechanical Turk platform.\n",
    "\n",
    "In 2010, the first ever ImageNet Large Scale Visual Recognition Challenge (ILSVRC) was organized. Software programs competed to correctly classify and detect objects and scenes.\n",
    "\n",
    "[Current record of the ImageNet (paperwithcode)](https://paperswithcode.com/sota/image-classification-on-imagenet)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## AlexNet\n",
    "\n",
    "AlexNet -- convolutional neural network that won the Imagenet challenge.\n",
    "\n",
    "The design of AlexNet and LeNet are very simple! (LeNet - left, AlexNet - right). Sigmoid is replaced by simple ReLU activations.\n",
    "\n",
    "What are the problems with this architecture?\n",
    "\n",
    "<img src=\"alexnet.svg\" width=\"40%\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## AlexNet: discussion\n",
    "\n",
    "Two problems: large convolutional filters in the beginning, and two large ML in the end ($6400 \\times 4096$ and $4096 \\times 4096$)\n",
    "\n",
    "The total number of trainable parameters is around $60M$ parameters.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## VGG\n",
    "\n",
    "In 2014, Simonyan and Zisserman proposed VGG (Visual Geometry Group) in Oxford network.\n",
    "\n",
    "**Key idea:** use multiple $3 \\times 3$ convolutions between MaxPooling downsampling.\n",
    "\n",
    "Receptive field of a $5 \\times 5$ convolution is similar to the receptive field of two $3 \\times 3$ convolutions,\n",
    "but the latter has less number of parameters! ($25^2$ vs $2 \\times 3^2$).\n",
    "\n",
    "They showed that deep and narrow convolutions outperform wider counterparts.\n",
    "\n",
    "3 x 3 convolutions became de-facto standard."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## VGG vs AlexNet\n",
    "\n",
    "The convolutions are grouped into the transformation that does not change the dimension, followed by dimension-reduction step.\n",
    "\n",
    "Original VGG had 5 blocks (first two have one conv. layer, others three).\n",
    "\n",
    "Alltogether, 8 convolutional layers and 3 fully-connected layers, hence the name **VGG-11**.\n",
    "\n",
    "VGG networks trained on ImageNet are excellent **feature extractors**.\n",
    "\n",
    "<img src=\"vgg.svg\" width=\"60%\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## VGG perceptual loss\n",
    "\n",
    "The pretrained VGG-19 network has very interesting features. If for two images those features are close,\n",
    "the images are similar.\n",
    "\n",
    "$$\\mathcal{L}_{VGG,(i, j)}(\\hat{y},y) = \\sum_{i,j} \\left\\|F_{ij}^l(\\hat{y}) - F_{ij}^l(y)\\right\\|_2^2$$\n",
    "\n",
    "Here $\\hat{y}$ is the reconstructed image, $y$ is the reference image, $F_{ij}^l(\\hat{y})$ and $F_{ij}^l(y)$ are the feature maps obtained by the $j$-th convolution (after activation) before the $i$-th maxpooling layer within the VGG19 network.\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## GoogLeNet \n",
    "\n",
    "The GoogLeNet model won ImageNet challenge in 2014 by using multibranch networks. \n",
    "It also has the design of low-level feature extractor (2-3 first layers), data processing and head (prediction).\n",
    "\n",
    "<img src=\"inception-full-90.svg\" width=\"60%\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Inception block\n",
    "\n",
    "The inception block has the following form\n",
    "\n",
    "<img src=\"inception.svg\" width=\"60%\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Training very deep network\n",
    "\n",
    "How we can increase the depth of the network without running into trouble?\n",
    "\n",
    "The answer is suprisingly simple and given by the **residual block**.\n",
    "\n",
    "Instead of mapping $x := f(x)$ we learn a mapping $x := x + f(x)$ trying to learn **correction** to the previous features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Residual block\n",
    "\n",
    "ResNet follows VGG design, but with residual blocks of the following form.\n",
    "\n",
    "<img src=\"residual-block.svg\" width=\"60%\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## ResNet: architecture\n",
    "\n",
    "The ResNet architecture is similar to GoogleNet, but is simpler and lead to wide adoption of such architectures.\n",
    "\n",
    "<img src=\"resnet18-90.svg\" width=\"60%\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Family of ResNet models\n",
    "\n",
    "Several modifications of the basic ResNet have been proposed:\n",
    "\n",
    "- ResNext: in the block, information flows through several groups and then aggregated (similar to the inception).\n",
    "- WideResnet: another block, shown to be superior\n",
    "\n",
    "<img src=\"wideresnet.png\" width=\"60%\">\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Typical properties of pretrained ResNet models\n",
    "\n",
    "Most popular (pretrained) models:\n",
    "\n",
    "| Model     | Top-1 Error | Top-5 Error | Number of Parameters |\n",
    "|-----------|-------------|-------------|----------------------|\n",
    "| ResNet-18 | 30.43       | 10.76       | 11,689,512           |\n",
    "| ResNet-34 | 26.73       | 8.74        | 21,797,672           |\n",
    "| ResNet-50 | 24.01       | 7.02        | 25,557,032           |\n",
    "| ResNet-101| 22.44       | 6.21        | 44,549,160           |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Factorization of convolutions\n",
    "\n",
    "Consider the convolutional layer:\n",
    "\n",
    "$$V(x, y, t) = \\sum_{i=x-\\delta}^{x+\\delta} \\sum_{j=y-\\delta}^{y+\\delta} \\sum_{s=1}^S K(i - x + \\delta, j - y + \\delta, s, t) U(i, j, s)$$\n",
    "\n",
    "Lets compute the following decomposition (called CP-decomposition) \n",
    "\n",
    "$$\n",
    "K(i,j,s,t) = \\sum_{r=1}^{R} K_x(i-x+\\delta,r) K_y(j-y+\\delta,r) K_s(s,r) K_t(t,r)\n",
    "$$\n",
    "\n",
    "Then the convolution is represented as $1 \\times 1$ convolution (which is just summation over the channel domain)\n",
    "\n",
    "and convolutions of smaller size!\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## MobileNet\n",
    "\n",
    "In 2017, a [MobileNet architecture](https://arxiv.org/pdf/1704.04861.pdf) has been proposed.\n",
    "\n",
    "The idea is to use $1 \\times 1$ convolutions plus depthwise separable convolutions (i.e., convolution is \n",
    "applied to each channel separatedly).\n",
    "\n",
    "What is the complexity of such transformation?\n",
    "\n",
    "The first architecture with depthwise separable convolutions has been used in 2016 by François Chollet (one of the authors of Keras) in the Inception block."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## MobileNetV1: architecture\n",
    "\n",
    "Left: VGG-type network, Right: MobileNet block.\n",
    "It has two parameters: width and and depth parameters (how many channels in the output, how much downsample).\n",
    "\n",
    "\n",
    "Also, there are no **pooling blocks**. \n",
    "Instead, the convolution with stride = 2 is used to reduce the dimension.\n",
    "\n",
    "<img src=\"mobilenet-block.png\" width=\"40%\">\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## MobileNetV2\n",
    "Note that one block has Residual connection, and the other does not have residual connection.\n",
    "Important: the first '1 x 1 convolution' increases the nubmer of channels, the last decreases. \n",
    "The same expansion block is used in transformers.\n",
    "\n",
    "<img src=\"mobilenetV2.png\" width=\"40%\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Some numbers from 2018\n",
    "\n",
    "| Network Architecture | Number of Parameters | Top-1 Accuracy | Top-5 Accuracy |\n",
    "|----------------------|----------------------|----------------|----------------|\n",
    "| Xception             | 22.91M               | 0.790          | 0.945          |\n",
    "| VGG16                | 138.35M              | 0.715          | 0.901          |\n",
    "| MobileNetV1 (alpha=1, rho=1) | 4.20M | 0.709 | 0.899 |\n",
    "| MobileNetV1 (alpha=0.75, rho=0.85) | 2.59M | 0.672 | 0.873 |\n",
    "| MobileNetV1 (alpha=0.25, rho=0.57) | 0.47M | 0.415 | 0.663 |\n",
    "| MobileNetV2 (alpha=1.4, rho=1) | 6.06M | 0.750 | 0.925 |\n",
    "| MobileNetV2 (alpha=1, rho=1) | 3.47M | 0.718 | 0.910 |\n",
    "| MobileNetV2 (alpha=0.35, rho=0.43) | 1.66M | 0.455 | 0.704 | \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## EfficientNet\n",
    "\n",
    "[Mingxing Tan 1 Quoc V. Le, EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](http://proceedings.mlr.press/v97/tan19a/tan19a.pdf)\n",
    "\n",
    "<div style=\"display: flex\">\n",
    "    <div style=\"flex: 1; margin-right:10px; width: 100%\">  \n",
    "<br />\n",
    "<br />The idea of scaling CNN architectures is very important.\n",
    "ConvNet layers are often partitioned into multiple\n",
    "stages and all layers in each stage share the same architecture: for example, \n",
    "\n",
    "ResNet (He et al., 2016) has **five stages,**\n",
    "and all layers in each stage has the same convolutional type\n",
    "\n",
    "except the first layer performs down-sampling. \n",
    "\n",
    "Therefore, we can define a ConvNet as:\n",
    "\n",
    "$\\mathcal{N} = \\bigodot_{i=1 \\ldots s} \\mathcal{F}^{L_i}_i (X_{\\langle H_i, W_i, C_i \\rangle}),$\n",
    "\n",
    "i.e. we have a layer applied $L_i$ times. \n",
    "        The idea of **EfficientNet** design is not to optimize $F_i$, but choose best $H_i, W_i, C_i$ and $L_i$.\n",
    "    </div>\n",
    "    <div style=\"flex: 1; margin-top: 20px\">\n",
    "        <img src=\"efficientnet.png\" width=\"80%\">\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Different types of scaling\n",
    "\n",
    "Typical scaling of ConvNet architectures:\n",
    "\n",
    "- Depth (but ResNet-1001 has similar accuracy to ResNet-100)\n",
    "- Width (WideResNets)\n",
    "- Resolution (higher input images).\n",
    "    <div style=\"flex: 1; margin-top: 20px\">\n",
    "        <img src=\"efficient-net-scaling.png\" width=\"100%\">\n",
    "    </div>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Compound scaling in EfficientNet\n",
    "\\begin{equation*}\n",
    "\\text{depth: } d = \\alpha^\\phi,\\quad\n",
    "\\text{width: } w = \\beta^\\phi,\\quad\n",
    "\\text{resolution: } r = \\gamma^\\phi,\n",
    "\\end{equation*}\n",
    "\\begin{equation*}\n",
    "\\text{s.t. } \\alpha \\cdot \\beta^2 \\cdot \\gamma^2 \\approx 2,\\quad \\alpha \\geq 1, \\beta \\geq 1, \\gamma \\geq 1.\n",
    "\\end{equation*}\n",
    "\n",
    "Parameter is determined by a grid search.\n",
    "\n",
    "The baseline model (EfficientNet-B0) for the block $\\mathcal{F}_i$ is determined by **Neural Architecture Search**.\n",
    "\n",
    "This idea of scaling basic blocks later emerged in transformers. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Other CNNs\n",
    "\n",
    "- **MobileNet** (the architecture with depth-wise separable convolutions, automatic search)\n",
    "- **DenseNet** (every layer is connected to all preceding onces)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Convolutions for other types of data\n",
    "\n",
    "We can have **1D convolutions** (sequences, audio-signals).\n",
    "\n",
    "**3D convNets**: for video data.\n",
    "\n",
    "Not very big difference in concepts compared to the 2D case (just the lack of ImageNet-type datasets)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## ConvNet for the 2020\n",
    "\n",
    "In 2020 the Visition Transformers (VIT) have outperformed CNN for many tasks. We will discuss VIT later. However, ConvNets can be modified to work on-par with VIT using several macro/micro design solution for the architecture.\n",
    "\n",
    "[ConvNext for the 2020 paper](https://arxiv.org/pdf/2201.03545.pdf)\n",
    "\n",
    "Key ideas:\n",
    "\n",
    "- Changing stem (the first block) to patchify. \n",
    "- Large kernel size (!!!!)  --- moving to $7 \\times 7$.\n",
    "- Inverted bottleneck\n",
    "- Several microdesign steps."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Important ingredients for better / stable training\n",
    "\n",
    "- **BatchNorm:** avoid gradient explosion\n",
    "- **Dropout:** reduce train/test gap.\n",
    "\n",
    "They will be covered in the next lecture in more details."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Take home message\n",
    "\n",
    "- Evolution of CNN\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Next lecture: Training better models\n",
    "\n",
    "- SGD optimization methods (SGD with momentum, Adam, ...)\n",
    "- Problems with training deep models: vanishing gradients, catastrophic forgetting\n",
    "- Effect of initializations\n",
    "- Normalizations \n",
    "- Early stopping \n",
    "- Interesting properties of loss surfaces of DNN models"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "celltoolbar": "Слайд-шоу",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "nav_menu": {},
  "rise": {
   "scroll": true,
   "theme": "sky",
   "transition": "zoom"
  },
  "toc": {
   "navigate_menu": true,
   "number_sections": false,
   "sideBar": true,
   "threshold": 6,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}