Workshop_plate_and_face_detection.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Kapernikov/workshop-face-and-plate-recognition/blob/main/Workshop_plate_and_face_detection.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pWFTo1ioyO4X"
      },
      "source": [
        "# Dependencies "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "E7PunryDg-8B"
      },
      "outputs": [],
      "source": [
        "!git clone https://github.com/Kapernikov/workshop-face-and-plate-recognition.git"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-oeNwYdOIDWQ"
      },
      "outputs": [],
      "source": [
        "!cd workshop-face-and-plate-recognition/ ; git pull"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eXGdE_cKyFf2"
      },
      "outputs": [],
      "source": [
        "!pip install -r workshop-face-and-plate-recognition/requirements.txt"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vHays9lo982V"
      },
      "outputs": [],
      "source": [
        "import cv2\n",
        "import onnxruntime as ort\n",
        "import numpy as np"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7ne6QzXPwhPQ"
      },
      "source": [
        "# Pre process\n",
        "Preprocessing is model dependant, the steps proposed here where done for the given models.\n",
        "\n",
        "We can see that model are size dependant which will be an issue while using the big images of the campaign (8000x4000) because we are gonna loose all details.\n",
        "\n",
        "The model is as well color models dependant, both use RGB here. And representation dependant. One model use [-1,1] to represent color while the other use [0,1]. These come from the images format use to train the model.\n",
        "We use explicite type to avoid an error of the automatic typer.\n",
        "\n",
        "The last necessary thing to do is to match the input shema, cv2 gives us [height, width, colors] and both model takes [image number, colors, height, width]."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5hLsAix_97jc"
      },
      "outputs": [],
      "source": [
        "def preprocess_for_face_detection(frame):\n",
        "    img = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\n",
        "    img = cv2.resize(img, (640, 480))\n",
        "    \n",
        "    from google.colab.patches import cv2_imshow\n",
        "    print(\"------ Input image for face detection  ------\")\n",
        "    cv2_imshow(img)\n",
        "    print(\"---------------------------------------------\")\n",
        "\n",
        "    img = (img - np.array([127, 127, 127])) / 128\n",
        "    img = np.transpose(img, [2, 0, 1])\n",
        "    img = np.expand_dims(img, axis=0)\n",
        "    img = img.astype(np.float32)\n",
        "\n",
        "    return img"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "iES0EJjnResS"
      },
      "outputs": [],
      "source": [
        "def preprocess_for_plate_detection(frame):\n",
        "    IN_IMAGE_H = ort_session_plate_model.get_inputs()[0].shape[2]\n",
        "    IN_IMAGE_W = ort_session_plate_model.get_inputs()[0].shape[3]\n",
        "\n",
        "    resized = cv2.resize(frame, (IN_IMAGE_W, IN_IMAGE_H), interpolation=cv2.INTER_LINEAR)\n",
        "    img_in = cv2.cvtColor(resized, cv2.COLOR_BGR2RGB)\n",
        "\n",
        "    from google.colab.patches import cv2_imshow\n",
        "    print(\"------ Input image for plate detection ------\")\n",
        "    cv2_imshow(img_in)\n",
        "    print(\"---------------------------------------------\")\n",
        "\n",
        "    img_in = np.transpose(img_in, (2, 0, 1)).astype(np.float32)\n",
        "    img_in = np.expand_dims(img_in, axis=0)\n",
        "    img_in /= 255.0\n",
        "    return img_in.astype(np.float32)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Grl1JxSgwmLR"
      },
      "source": [
        "# Run the model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nBZUI4P4-Nnx"
      },
      "source": [
        "## Models output format\n",
        "- boxes: a list of pair of points defining boxe where object have been detected.\n",
        "- confidences: contains a list of confidence level for each box inside the boxes variable. The first and second values of one confidence pair indicate the probability of containing background and face respectively. It means that our model have two classes that he can detect, namely background and face, and give the probability for each of them."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fWVFHHzMdIrR"
      },
      "source": [
        "## Face detection model (pre-trained)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bXrDDHC6-MYH"
      },
      "outputs": [],
      "source": [
        "ort_session_face_model = ort.InferenceSession(\"\"\"workshop-face-and-plate-recognition//models//ultra_light_640.onnx\"\"\")\n",
        "input_name_face_model = ort_session_face_model.get_inputs()[0].name"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "saIKiHBI-H-m"
      },
      "outputs": [],
      "source": [
        "def predict_face_area(input_data):\n",
        "    confidences, boxes = ort_session_face_model.run(None, {input_name_face_model: input_data})\n",
        "    return confidences, boxes"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CWT64oIedPGm"
      },
      "source": [
        "## License plate detection model (pre-trained)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6jqcgd7CSVg2"
      },
      "outputs": [],
      "source": [
        "# This model doesn't fit on the git\n",
        "!gdown 16NPM0SGP-p6R3VNb9eB6rqM7-E2-EjKX -O workshop-face-and-plate-recognition/models/yolov4_1_3_416_416_static.onnx"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "C-5ORJr9R1xY"
      },
      "outputs": [],
      "source": [
        "#ort_session_plate_model = ort.InferenceSession(\"\"\"/content/drive/MyDrive/onnx_model/yolov4_1_3_416_416_static.onnx\"\"\")\n",
        "ort_session_plate_model = ort.InferenceSession(\"\"\"/content/workshop-face-and-plate-recognition/models/yolov4_1_3_416_416_static.onnx\"\"\")\n",
        "input_name_plate_model = ort_session_plate_model.get_inputs()[0].name"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QEousacoVm67"
      },
      "outputs": [],
      "source": [
        "def predict_plate(input_data):\n",
        "    boxes, confidences = ort_session_plate_model.run(None, {input_name_plate_model: input_data})\n",
        "    return confidences, boxes[:,:,0,:]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xc8Pv1mkwqNt"
      },
      "source": [
        "# post process"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ftKZ8IILhMtR"
      },
      "source": [
        "1. For each box, we want to define its class. Therefor, for each class (except the background one), we will take the confidence index that are above a threshold:\n",
        "    \tprobs = confidences[:, class_index]\n",
        "    \tmask = probs > prob_threshold  # return boolean array that is True if the probability is above threshold, else it is False\n",
        "    \tprobs = probs[mask]\n",
        "    \tsubset_boxes = boxes[mask, :]\n",
        "\n",
        "What class do we have?\n",
        "\n",
        "\n",
        "The first model gives us the probability of being part of the surrounding and the probability of being a face, the second model gives us the probability of being a license plate. This gives un 3 classes."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fUcg0MXeg64C"
      },
      "source": [
        "2. On each of these boxe/probability pair we will perform a non-maximum-suppression, therefore we will define a intersection-over-union function as a function returning the overlapping area / non-overlapping area\n",
        "\n",
        "To perform the actual non-maximum-suppression, we will:\n",
        "- select the remaining candidate with the highest score\n",
        "- remove the candidate with an intersection-over-union above a threshold\n",
        "- until there are no remaining candidates\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TBkWokaF-orB"
      },
      "outputs": [],
      "source": [
        "def get_area_of(left_top, right_bottom):\n",
        "    hw = np.clip(right_bottom - left_top, 0.0, None)\n",
        "    return hw[..., 0] * hw[..., 1]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WSV5qzv2-t7L"
      },
      "outputs": [],
      "source": [
        "def get_iou_of(boxes0, boxes1):\n",
        "    overlap_left_top = np.maximum(boxes0[..., :2], boxes1[..., :2])\n",
        "    overlap_right_bottom = np.minimum(boxes0[..., 2:], boxes1[..., 2:])\n",
        "\n",
        "    overlap_area = get_area_of(overlap_left_top, overlap_right_bottom)\n",
        "    area0 = get_area_of(boxes0[..., :2], boxes0[..., 2:])\n",
        "    area1 = get_area_of(boxes1[..., :2], boxes1[..., 2:])\n",
        "    return overlap_area / (area0 + area1 - overlap_area + 1e-5)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Mz8Fc7F_-4Pc"
      },
      "outputs": [],
      "source": [
        "def proceed_hard_nms(box_scores, iou_threshold, top_k=-1, candidate_size=200):\n",
        "    scores = box_scores[:, -1]\n",
        "    boxes = box_scores[:, :-1]\n",
        "    picked = []\n",
        "    indexes = np.argsort(scores)\n",
        "    indexes = indexes[-candidate_size:]\n",
        "    while len(indexes) > 0:\n",
        "        current = indexes[-1]\n",
        "        picked.append(current)\n",
        "        if 0 < top_k == len(picked) or len(indexes) == 1:\n",
        "            break\n",
        "        current_box = boxes[current, :]\n",
        "        indexes = indexes[:-1]\n",
        "        rest_boxes = boxes[indexes, :]\n",
        "        iou = get_iou_of(\n",
        "            rest_boxes,\n",
        "            np.expand_dims(current_box, axis=0),\n",
        "        )\n",
        "        indexes = indexes[iou <= iou_threshold]\n",
        "\n",
        "    return box_scores[picked, :]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eg_wkw1MARiQ"
      },
      "outputs": [],
      "source": [
        "def postprocess(width, height, confidences, boxes, prob_threshold, iou_threshold=0.5, top_k=-1, skip_n_class=0):\n",
        "    boxes = boxes[0]\n",
        "    confidences = confidences[0]\n",
        "    picked_box_probs = []\n",
        "    picked_labels = []\n",
        "    for class_index in range(skip_n_class, confidences.shape[1]):\n",
        "        probs = confidences[:, class_index]\n",
        "        mask = probs > prob_threshold\n",
        "        probs = probs[mask]\n",
        "        if probs.shape[0] == 0:\n",
        "            continue\n",
        "        subset_boxes = boxes[mask, :]\n",
        "        box_probs = np.concatenate([subset_boxes, probs.reshape(-1, 1)], axis=1)\n",
        "        box_probs = proceed_hard_nms(box_probs,\n",
        "           iou_threshold=iou_threshold,\n",
        "           top_k=top_k,\n",
        "           )\n",
        "        picked_box_probs.append(box_probs)\n",
        "        picked_labels.extend([class_index] * box_probs.shape[0])\n",
        "    if not picked_box_probs:\n",
        "        return np.array([]), np.array([]), np.array([])\n",
        "    picked_box_probs = np.concatenate(picked_box_probs)\n",
        "    picked_box_probs[:, 0] *= width\n",
        "    picked_box_probs[:, 1] *= height\n",
        "    picked_box_probs[:, 2] *= width\n",
        "    picked_box_probs[:, 3] *= height\n",
        "    return picked_box_probs[:, :4].astype(np.int32), np.array(picked_labels), picked_box_probs[:, 4]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CHhQRzdxwvHn"
      },
      "source": [
        "# Process zone of interest\n",
        "\n",
        "## Blur the bounding boxes \n",
        "**several methods:**\n",
        "1. cv2.blur() will apply a kernel on the desired portion of the image to blur it. BUT as it use a mathematical function to do it, it might be reversible.\n",
        "\n",
        "2. Set the color to a fix value in the region of interest. Remove the information completely!\n",
        "\n",
        "3. Interpolate the region of interest from a very low resolution\n",
        "Remove part of the information, the lower the resolution is the more distant the final result will be from the original image\n",
        "Smoother than the fixed color rectangle\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "d-F5Rn-Tw5z1"
      },
      "outputs": [],
      "source": [
        "def blur_box(frame, box):\n",
        "    h, w, _ = frame.shape\n",
        "    x1, y1, x2, y2 = box\n",
        "\n",
        "    roi_buffer = max((x2-x1), (y2-y1))//4\n",
        "    x1_buffered = max(0, x1-roi_buffer)\n",
        "    y1_buffered = max(0, y1-roi_buffer)\n",
        "    x2_buffered = min(w, x2+roi_buffer)\n",
        "    y2_buffered = min(h, y2+roi_buffer)\n",
        "\n",
        "    roi = frame[y1_buffered:y2_buffered, x1_buffered:x2_buffered]\n",
        "    roi = cv2.resize(roi, (8, 8), interpolation=cv2.INTER_NEAREST)\n",
        "    roi = cv2.resize(roi, (x2_buffered-x1_buffered, y2_buffered-y1_buffered), interpolation=cv2.INTER_NEAREST)\n",
        "    frame[y1_buffered:y2_buffered, x1_buffered:x2_buffered] = roi"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1bNgEpO0w8qL"
      },
      "outputs": [],
      "source": [
        "def draw_box(frame, box, label):\n",
        "    h, w, _ = frame.shape\n",
        "    x1, y1, x2, y2 = box\n",
        "\n",
        "    roi_buffer = max((x2-x1), (y2-y1))//4\n",
        "    x1_buffered = max(0, x1-roi_buffer)\n",
        "    y1_buffered = max(0, y1-roi_buffer)\n",
        "    x2_buffered = min(w, x2+roi_buffer)\n",
        "    y2_buffered = min(h, y2+roi_buffer)\n",
        "\n",
        "    cv2.rectangle(frame, (x1, y1), (x2, y2), (80,18,236), 2)\n",
        "    cv2.rectangle(frame, (x1, y2 - 20), (x2, y2), (80,18,236), cv2.FILLED)\n",
        "    font = cv2.FONT_HERSHEY_DUPLEX\n",
        "    text = f\"label: {label}\"\n",
        "    cv2.putText(frame, text, (x1 + 6, y2 - 6), font, 0.5, (255, 255, 255), 1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XsNgfbwfAi1_"
      },
      "source": [
        "# Glue all together"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "s18s34xlBrjV"
      },
      "outputs": [],
      "source": [
        "!wget -c https://media.wired.com/photos/5926f91dcfe0d93c47431f76/master/w_1600%2Cc_limit/Top-Gear-Series-23-Preview_11_35MB.jpg -O image_test.jpg"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4tA1zuGxAnPV"
      },
      "outputs": [],
      "source": [
        "file_name = \"image_test.jpg\"\n",
        "video_capture = cv2.VideoCapture(file_name)\n",
        "\n",
        "ret, frame = video_capture.read()\n",
        "\n",
        "input_data = preprocess_for_face_detection(frame)\n",
        "input_data_plate = preprocess_for_plate_detection(frame)\n",
        "\n",
        "confidences_face, boxes_face = predict_face_area(input_data)\n",
        "confidences_plate, boxes_plate = predict_plate(input_data_plate)\n",
        "\n",
        "boxes = np.concatenate((boxes_face, boxes_plate), axis=1)\n",
        "confidences = np.concatenate((np.insert(confidences_face, 2, 0, axis=2), np.insert(confidences_plate, (0,0), 0, axis=2)), axis=1)\n",
        "\n",
        "h, w, _ = frame.shape\n",
        "boxes, labels, probs = postprocess(w, h, confidences, boxes, 0.5, skip_n_class=1)\n",
        "\n",
        "frame_result_1 = np.array(frame)\n",
        "frame_result_2 = np.array(frame)\n",
        "\n",
        "for i in range(boxes.shape[0]):\n",
        "    box = boxes[i, :]\n",
        "    blur_box(frame_result_2, box)\n",
        "    draw_box(frame_result_1, box, labels[i])\n",
        "\n",
        "# cv2.imshow('Image', frame)\n",
        "from google.colab.patches import cv2_imshow\n",
        "print(\"------         Original image         ------\")\n",
        "cv2_imshow(frame)\n",
        "print(\"------         Anotated image         ------\")\n",
        "cv2_imshow(frame_result_1)\n",
        "print(\"------         blurred  image         ------\")\n",
        "cv2_imshow(frame_result_2)\n",
        "print(\"--------------------------------------------\")\n",
        "\n",
        "video_capture.release()\n",
        "cv2.destroyAllWindows()"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "mount_file_id": "1hrDFw1AEEmTcTRHeFaBzvWCay1QlMavF",
      "authorship_tag": "ABX9TyONNo1QNeb6rWmyJhTTrBDb",
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}