lecture_19_object-detection.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## VGG-16"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Before we start describing SSD model let's have a quick look on VGG-16 architecture which is used as a backbone or feature extractor for SSD model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Model VGG (visual geometry group) was first presented at 2013 ImageNet competition and got high results (not winner ZENet was winner) but it was fast and had a simple architecture.\n",
    "VGG-16 is still used in\n",
    "- Backbone for many models\n",
    "- Perceptual loss for style transfer and auto-encoder (super-resolution, variational auto-encoder) models\n",
    "- From my experience, VGG-16 has a place, color and size invariant feature extraction property because of pooling layers and convolutional stack."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "VGG-16 architecture:\n",
    "<img src=\"images/od/vgg_16_1.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "One of the best features extractor for background jobs:\n",
    "- High dimension\n",
    "- Spark job for image clustering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## SSD (Single shot multi box detection)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "SSD is one of the single shot detection family members.\n",
    "<br>\n",
    "For object detection, as we remember from localization models, we need fixed number of bounding boxes.\n",
    "<br>\n",
    "SSD does this by assigning $n$ bounding boxes per pixel on feature map\n",
    "<br>\n",
    "Why on features map, because they have smaller number of pixels, smaller special (height and with) size\n",
    "<br>\n",
    "The number of bounding boxes might be different for the layer\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "General architecture with non max suppression:\n",
    "<img src=\"images/od/ssd_2.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Detector and classifier block:\n",
    "<img src=\"images/od/ssd_3.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We have a feature map with low spatial size for block, generate three bounding boxes per pixel:\n",
    "<img src=\"images/od/ssd_4.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "With corner correction:\n",
    "<img src=\"images/od/ssd_7.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Train with appropriated loss:\n",
    "<img src=\"images/od/ssd_10.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Different amount of bounding boxes per layer and correction at the end:\n",
    "<img src=\"images/od/ssd_12.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## ResNet and notion of the skip-connections"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "ResNet by the Microsoft was winner of the 2015 ImageNet competition and the first model which outperformed the human"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The main idea behind the ResNet architecture is so called skip-connections or residual blocks:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Residual block:\n",
    "<img src=\"images/od/resnet_1.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "With different interpretations:\n",
    "<img src=\"images/od/resnet_2.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "With the skip connections models models depth limit was increased with performance gain"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "ResNet vs other models\n",
    "<img src=\"images/od/resnet_3.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Almost all modern architectures for classification (which are used as backbones), or encoders use some kind of modification of residual connections:\n",
    "- Ineption-ResNet\n",
    "- ResNeXt\n",
    "- MobileNet\n",
    "- EfficientNet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Residual connections in other architectures:\n",
    "<img src=\"images/od/resnet_4.jpg\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Feature pyramid models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Lets combing down-sampling and up-sampling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "UNet\n",
    "<img src=\"images/od/unet.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's make it slightly faster"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/od/fpn_1.jpeg\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/od/fpn_2.jpeg\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## RetinaNet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Feature pyramids with ResNet50 as backbone:\n",
    "\n",
    "<img src=\"images/od/retina_1.jpg\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Feature pyramid extracts and preserves semantically and spatially, after some feature pyramid block (two) predictors are applied, bounding box regression with $H \\times W$ spacial size and $4A$ channels and classification networks with $H \\times W$ spacial size and $KA$ channels, class per anchor box:\n",
    "\n",
    "<img src=\"images/od/retina_2.jpg\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "And classification:\n",
    "<img src=\"images/od/retina_3.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Regression loss:\n",
    "$$\n",
    "\\begin{align}\n",
    "T^i_x &= (G^i_x - A^i_x) / A^i_w  \\\\\n",
    "T^i_y &= (G^i_y - A^i_y) / A^i_h \\\\\n",
    "T^i_w &= log(G^i_w / A^i_w) \\\\\n",
    "T^i_h &= log(G^i_h / A^i_h)\n",
    "\\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "And loss is applied as:\n",
    "$$\n",
    "L_{loc} = \\sum_{j \\in \\{x, y, w, h\\}}smooth_{L1}(P^i_j - T^i_j)\n",
    "$$\n",
    "<br>\n",
    "where:\n",
    "$$\n",
    "\\begin{equation}\n",
    "smooth_{L1}(x) = \n",
    "\\begin{cases}\n",
    "0.5x^2 &|x| < 1 \\\\\n",
    "|x| - 0.5 &|x| \\geq 1\n",
    "\\end{cases}\n",
    "\\end{equation}\n",
    "$$\n",
    "<br>\n",
    "and\n",
    "$$\n",
    "P^i = (P^i_x, P^i_y, P^i_w, P^i_h)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Focal loss:\n",
    "$$\n",
    "L_{cls} = -\\sum_{i=1}^{K}(y_ilog(p_i)(1-p_i)^\\gamma \\alpha_i + (1 - y_i)log(1 - p_i)p_i^\\gamma (1 - \\alpha_i))\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Here we have two dependent hyper-parameters:\n",
    "- Weighting parameter $\\alpha$ which is responsible for class imbalance $\\alpha_i \\in [0, 1]$\n",
    "- Focusing parameter $\\gamma$ for background foreground distinguish $\\gamma \\in (0, +\\infty)$\n",
    "- During the weight initialization, biases on the last layer are initialized bigger with some rules"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "This parameters are balancing for meaning hard negatives.\n",
    "<br>\n",
    "Because of big amount of background images, data might be classified as background easily"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Here's the example of training with and without focal loss:\n",
    "<img src=\"images/od/focal_1.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "At the end we'll add that each detector outputs $1$K boxes and then reduced by the non max suppression algorithm "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Multi-shot object detection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Problems with detecting small objects\n",
    "- Different size of sliding windows\n",
    "- Run classifier network per patch\n",
    "- Computationally expensive"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Region proposals"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "With classical CV (greedy) algorithm calculated regions of images - Selective Search\n",
    "<img src=\"images/od/selective_search.jpg\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### R-CNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We calculate 2000 region proposals and run them through the pre-trained cnn classifier (VGG-16 or ResNet-50)\n",
    "<img src=\"images/od/r_cnn_1.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We train SVM classifiers per extracted features and also BBox regression to improve boxes:\n",
    "<img src=\"images/od/r_cnn_2.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Improves accuracy\n",
    "- Computationally expensive for training\n",
    "- Computationally expensive for testing / inference\n",
    "- Slow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Fast-RCNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Instead of calculation of region proposals on input image, we first run image through CNN model and calculate region proposals on smaller feature map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Region proposals have high recall and many of them are classified as background"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Region proposals $N \\times 5$ image index and coordinates\n",
    "<img src=\"images/od/region_proposal_cat.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Then we use ROI pooling to adjust features and flatten pooled image parts for classification\n",
    "ROI poolint takes a section of the input feature map that corresponds to it and scales it to some pre-defined size\n",
    "<img src=\"images/od/roi_pooling_1.gif\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/od/fast_r_cnn.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Less computational resources\n",
    "- Faster training time\n",
    "- Faster testing / inference time\n",
    "- The same performance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Faster-RCNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Instead of using selective search, let's use separate convolutional neural network which calculates region proposals"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/od/faster_r_cnn_1.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/od/faster_r_cnn_2.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Less computationally expensive\n",
    "- Region proposals calculation is learnable\n",
    "- Faster for training\n",
    "- Faster for test / inference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Questions\n",
    "<img src=\"images/od/questions_2.jpg\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Thank you"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}