lecture_18_segmentation_detection.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Segmentation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Image patch classifier\n",
    "<img src=\"images/od/semantic_segmentation_sliding_window.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Computationally expensive"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Image patch classifier\n",
    "<img src=\"images/od/semantic_segmentation_sliding_window.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Computationally expensive and hard to classify without additional information of image context"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Fully convolutional model byt at the end layers information is lost\n",
    "<img src=\"images/od/fully_conv.jpeg\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Fully convolutional model size preserving\n",
    "<img src=\"images/od/fully_convolutional_2.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Without down-sampling preserve original image scale would be really expensive and that's why encoder-decoder model is preferred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Down-sampling plus up-sampling (Encoder decoder architecture)\n",
    "<img src=\"images/od/encoder_decoder_1.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Encoder-decoder application:\n",
    "<img src=\"images/od/encoder_decoder_2.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Detects smaller objects on first layers of encoder and last layers on decoder\n",
    "- Detects bigger objects on last layers of encoder and fist layers on decoder\n",
    "- Decodes medium sized objects on medium layers\n",
    "- Example auto-encoders"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/od/autoencoder.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Up-sampling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Unpooling example (non-learnable):\n",
    "<img src=\"images/od/unpooling_1.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Recall convolution with stride:\n",
    "<img src=\"images/od/conv_stride_1.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Transpose convolution:\n",
    "<img src=\"images/od/transpose_1.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Transpose convolution for $1$ dimension:\n",
    "<img src=\"images/od/transpose_2.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "After each layer naturally there are activations, maybe normalization and other layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "If we write layer as matrix operation:\n",
    "<img src=\"images/od/transpose_matrix_1.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Transpose convolution because of it's nature of repeating the same value from receptive field, causes the so called \"checkerboard\" artifacts:\n",
    "<img src=\"images/od/checkerboard_1.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Other up-sampling methods:\n",
    "Up-sample with chosen (bilinear) interpolation for instance twice or three times and then run convolution:\n",
    "<br>\n",
    "$$\n",
    "I_r = \\frac{I - F_d + 2 P}{S_d} + 1 = \\frac{I}{S_d} - \\frac{F_d - 2 P}{S_d} + 1\n",
    "$$\n",
    "<br>\n",
    "so we get:\n",
    "$$\n",
    "I_r = \\frac{n \\cdot I - F_d + 2 P}{S_d} + 1 = n \\cdot \\frac{I}{S_d} - \\frac{I}{F_d - 2 P}{S_d} + 1\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "There are bunch of other learnable up-sampling methods, for instance pixel shuffle up-sampling instead of other interpolation before convolution layer:\n",
    "<img src=\"images/od/pixel_shuffle_1.jpg\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The main results are:\n",
    "- Up-sample might be learnable layer\n",
    "- It's pretty similar to convolution layer with activations \n",
    "- Has the same depth for filter as input and outputs no depth\n",
    "- Might generate as match up-sampled feature maps as we configure"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "During the down-sample convolutional layers, information is extracted from the input image independent from the tasks sometimes.\n",
    "<br>\n",
    "Often pre-trained (on ImageNet classification task) encoder (often called backbone) used"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For segmentation we can use $L_2$ or $L_1$ loss:\n",
    "$$\n",
    "L_2 = \\sum_{i = 1}^{h}\\sum_{j=1}^{w}||y_{ij} - \\hat{y}_{ij}||^2\n",
    "$$\n",
    "<br>\n",
    "$$\n",
    "L_1 = \\sum_{i = 1}^{h}\\sum_{j=1}^{w}||y_{ij} - \\hat{y}_{ij}||\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can produce as many feature maps on last layer as many objects we have\n",
    "<br>\n",
    "or different number can denote different objects for labels:\n",
    "$$\\begin{align} Y &= \\begin{pmatrix}\n",
    "           0, 1, \\dots, 0 \\\\\n",
    "           1, 1, \\dots, 0 \\\\\n",
    "           \\vdots \\\\\n",
    "           0, 1, \\dots, 1 \\\\\n",
    "         \\end{pmatrix}\n",
    "  \\end{align}$$\n",
    "  <br>\n",
    "  or:\n",
    "  $$\\begin{align} Y &= \\begin{pmatrix}\n",
    "           1, 2, \\dots, 4 \\\\\n",
    "           0, 4, \\dots, 2 \\\\\n",
    "           \\vdots \\\\\n",
    "           2, 1, \\dots, 0 \\\\\n",
    "         \\end{pmatrix}\n",
    "  \\end{align}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Why not mix encoder-decoder models in segmentation-recognition models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Mix encoder-decoder model layers\n",
    "<img src=\"images/od/mixing_layers_1.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## UNet combine encoder and decoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Lets combing down-sampling and up-sampling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "UNet\n",
    "<img src=\"images/od/unet.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Lets combing down-sampling and up-sampling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "UNet\n",
    "<img src=\"images/od/unet_2.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Single shot detection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Multi model has high performance but at the same time:\n",
    "- they are hard to train\n",
    "- hard to select correct parameters / hyper-parameters\n",
    "- computationally expensive during the training\n",
    "- computationally expensive during the inference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For the real time or near-real time approaches there exists different single shot detectors\n",
    "Which runs bounding box regression and the classification at the same time"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Recall localization and classification algorith:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/od/localization_classification.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In some aspects, this is the main approach, we can not have a undefined output of the feed-forward neural networks (not RNN)\n",
    "<br>\n",
    "So we need somehow localize single object in the image\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "First approach which we saw in the previous presentation was the family of the RCNN models\n",
    "<br>\n",
    "Model first detected blobs with different algorithms (Region proposals, Region proposals on the feature map, decoder encoder-model) and then uses classification + localization algorithm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "On the other hand we can split the image by the smaller parts for instance $10 \\times 10$ or $6 \\times 6$ or even $19 \\times 19$ parts, grid-cells"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/od/yolo_1.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For the object detection we have:\n",
    "- Object classes: $(c_1, c_2, \\dots, c_n)$\n",
    "- Bounding boxes: $((x, y, h, w )_1, (x, y, h, w )_2 \\dots (x, y, h, w )_m)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "And for each grid cell (which is small enough) lets take center and classify is there object or not:\n",
    "- $p$ is the probability is there object or not\n",
    "- Also class label $𝑐_1,𝑐_2,…,𝑐_𝑛$\n",
    "- Bounding boxes $x, y, h, w $ for this instant class"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "So we have:\n",
    "$$\n",
    "\\begin{align}\n",
    "    g_i &= \\begin{bmatrix}\n",
    "           p \\\\\n",
    "           x \\\\\n",
    "           y \\\\\n",
    "           h \\\\\n",
    "           w \\\\\n",
    "           c_1 \\\\\n",
    "           c_2 \\\\\n",
    "           \\vdots \\\\\n",
    "           c_n\n",
    "         \\end{bmatrix}\n",
    "  \\end{align}\n",
    "$$\n",
    "<br>\n",
    "For each grid-cell of image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now let consider it as a tensor with $6 \\times 6$, $14 \\times 14$, $19 \\times 19$, $g \\times g$ or any other, with $g_i$ grid cell vector for each\n",
    "<br>\n",
    "$g \\times g \\times (5 + c)$\n",
    "- So use this as a target for the prediction and run model per grid-cell\n",
    "- Or we can use compose our model with the single model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "So lets create neural network which has a convolutional layers with the our $g \\times g$ model\n",
    "<img src=\"images/od/yolo_2.gif\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## YOLO-v3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Previous YOLO models does not have up-sampling architectures which means that it was fast but it was not accurate enough\n",
    "<br>\n",
    "In YOLOv3 was added DarkNet53 backbone model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "YOLOv3 model architecture with upsampling layers:\n",
    "<img src=\"images/od/yolo_v3_arch.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Anchor boxes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Recall that YOLOv3 was detecting objects for each cell and was assigning \n",
    "$$  \\begin{align}\n",
    "    y &= \\begin{pmatrix}\n",
    "           p \\\\\n",
    "           x \\\\\n",
    "           y \\\\\n",
    "           h \\\\\n",
    "           w \\\\\n",
    "           c_{1} \\\\\n",
    "           c_{2} \\\\\n",
    "           \\vdots \\\\\n",
    "           c_{n}\n",
    "         \\end{pmatrix}\n",
    "  \\end{align}\n",
    "$$\n",
    "<br>\n",
    "Let's imagine the scenario where two objects have the same center. Then our detection should decide by probability. But what if both of them are in the image?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Per cell we had a sigle vector:\n",
    "<img src=\"images/od/yolo_pred_1.jpeg\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For instance person and car have the same center (no the best place for center):\n",
    "<img src=\"images/od/anchor_1.jpeg\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Or the same here:\n",
    "<img src=\"images/od/anchor_2.jpg\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "If we observe the shape of the objects, the car of them has bigger width and smaller height while the person has smaller width and bigger height\n",
    "<br>\n",
    "Generally different type if objects has different shape and size and they might be assigned to the detection algorithm\n",
    "<img src=\"images/od/anchor_3.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's use this information and add two predefined shapes to our algorithm, instead of one vector per cell, let's concatenate two of them, one pre-shape which we'll call anchor box:\n",
    "$$  \\begin{align}\n",
    "    y_1 = \\begin{pmatrix}\n",
    "           p \\\\\n",
    "           x \\\\\n",
    "           y \\\\\n",
    "           h \\\\\n",
    "           w \\\\\n",
    "           c_{1} \\\\\n",
    "           c_{2} \\\\\n",
    "           \\vdots \\\\\n",
    "           c_{n}\n",
    "         \\end{pmatrix}\n",
    "\\quad\n",
    "    y_2 = \\begin{pmatrix}\n",
    "           p \\\\\n",
    "           x \\\\\n",
    "           y \\\\\n",
    "           h \\\\\n",
    "           w \\\\\n",
    "           c_{1} \\\\\n",
    "           c_{2} \\\\\n",
    "           \\vdots \\\\\n",
    "           c_{n}\n",
    "         \\end{pmatrix}\n",
    "  \\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Concatenate them and we will get:\n",
    "$$ \\begin{align}\n",
    "    y = \\begin{pmatrix}\n",
    "           p \\\\\n",
    "           x \\\\\n",
    "           y \\\\\n",
    "           h \\\\\n",
    "           w \\\\\n",
    "           c_{1} \\\\\n",
    "           c_{2} \\\\\n",
    "           \\vdots \\\\\n",
    "           c_{n} \\\\\n",
    "           p \\\\\n",
    "           x \\\\\n",
    "           y \\\\\n",
    "           h \\\\\n",
    "           w \\\\\n",
    "           c_{1} \\\\\n",
    "           c_{2} \\\\\n",
    "           \\vdots \\\\\n",
    "           c_{n}\n",
    "         \\end{pmatrix}\n",
    "  \\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Or maybe we'll group probabilities, together or probabilities and bounding boxes, etc or put in other order, it does not matter:\n",
    "$$ \\begin{align}\n",
    "    y = \\begin{pmatrix}\n",
    "           x \\\\\n",
    "           y \\\\\n",
    "           h \\\\\n",
    "           w \\\\\n",
    "           c_{1} \\\\\n",
    "           c_{2} \\\\\n",
    "           \\vdots \\\\\n",
    "           c_{n} \\\\\n",
    "           x \\\\\n",
    "           y \\\\\n",
    "           h \\\\\n",
    "           w \\\\\n",
    "           c_{1} \\\\\n",
    "           c_{2} \\\\\n",
    "           \\vdots \\\\\n",
    "           c_{n} \\\\\n",
    "           p_1 \\\\\n",
    "           p_2\n",
    "         \\end{pmatrix}\n",
    "  \\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We will have $19 \\times 19 \\times 2 \\times {(1 + 2 + 4 + n)}$ dimensional output\n",
    "<br>\n",
    "Or as tensor $19 \\times 19 \\times {2 \\cdot (1 + 2 + 4 + n)} = 19 \\times 19 \\times {14 + 2 n}$\n",
    "<img src=\"images/od/yolo_2.gif\" height=\"600\" width=\"600\">\n",
    "This is the answer of the question from the previous presentation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In practice there are more than two anchor boxes for detection models:\n",
    "<img src=\"images/od/pred_anchor_1.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Actually anchor boxes are predefined and used as default during the training.\n",
    "During the training annotated object and it's bounding box is assigned to the concrete anchor box by the highest IoU value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "People might generate anchor boxes by hand or use KNN or other advanced techniques for this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The main flaw of anchor boxes is if there are more object with the same center that anchor boxes.\n",
    "<br>\n",
    "This is the case which anchor boxes approach can not deal with\n",
    "<br>\n",
    "But usually there are many enough anchor boxes and small enough cells in YOLO case to make this event almost impossible\n",
    "<br>\n",
    "For instance $19 \\times 19$ cells are small enough with $9$ anchor boxes to avoid $9$ different object with the same center"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now the anchor boxes are used in many detection algorithms. Just increase output value by the fixed number"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Non-max suppression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Generally detector models predict multiply bounding boxes per object (per image) for instance YOLO might consider several several cells as center of the instant object:\n",
    "<img src=\"images/od/non_max_1.jpg\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can filter them by confidence and IoU thresholds, but still there will be many of them:\n",
    "<img src=\"images/od/non_max_2.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Also, there might be many same category of object on a single image. How can we reduce number of boxes:\n",
    "<br>\n",
    "For a single object category:\n",
    "- First remove all boxes with lower confidence that threshold\n",
    "- Second we choose box with highest confidence\n",
    "- Then calculate IoU with other boxes and if IoU is higher than $0.5$ for instance, remove this box\n",
    "- Then do the same with the rest of the boxes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Remove low confidence, sort and then run non max suppression:\n",
    "<img src=\"images/od/non_max_det_1.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Leave only low IoU box scores others put $0$ values and filter\n",
    "<img src=\"images/od/non_max_det_2.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "If we have a single object of the category:\n",
    "<img src=\"images/od/non_max_3.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "But why do we leave lower IoU images?\n",
    "<br>\n",
    "Because they might be the boxes for the different object of the same category:\n",
    "<img src=\"images/od/non_max_4.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "After we remove boxes with higher IoU we'll distinguish different objects of the same category:\n",
    "<img src=\"images/od/non_max_5.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Here's the other example of different of the same category:\n",
    "<img src=\"images/od/non_max_7.jpeg\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Multi-centers:\n",
    "<img src=\"images/od/non_max_8.jpg\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Questions\n",
    "<img src=\"images/od/questions_1.jpg\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Thank you"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.2"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}