notebooks/8_configs.ipynb

{
 "cells": [
  {
   "cell_type": "raw",
   "id": "b090fbbf-d8e5-4e48-b87c-4f09a121995b",
   "metadata": {},
   "source": [
    "---\n",
    "title: Running models with your own data\n",
    "author: Janne Mäyrä\n",
    "date: last-modified\n",
    "format:\n",
    "  html:\n",
    "    code-fold: true\n",
    "    toc: false\n",
    "    toc-depth: 3\n",
    "    number-sections: true\n",
    "    smooth-scroll: true\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "171ddd64-0b57-46dc-81da-6e224b28d08f",
   "metadata": {},
   "source": [
    "All model configurations and weights here are fine-tuned from models available in [Detectron2 Model Zoo](https://github.com/facebookresearch/detectron2/blob/main/MODEL_ZOO.md).\n",
    "\n",
    "The models are trained on 512x512px RGB image patches with spatial resolution between 3.9cm and 4.3 cm and with hand-annotated deadwood data based on visual inspection. The location of training dataset is in the vicinity of Hiidenportti national, Sotkamo, Finland, and the images were acquired during leaf-on season, on 16. and 17.7.2019. Most likely the models are most suitable to use with imagery from leaf-on season and with similar ground sampling distance."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05eb8122-fdce-4a67-9847-2343fd545e80",
   "metadata": {},
   "source": [
    "# Available models and results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3d8f211-ab36-49b3-9ba5-5bf3d86b74ca",
   "metadata": {},
   "source": [
    "Model configs and weights are available [here](https://huggingface.co/mayrajeo/maskrcnn-deadwood). Note that configs for `WEIGHTS` and `OUTPUT_DIR` need to be changed according to your needs. These models are trained only with Hiidenportti dataset. Example app running R101 without TTA for image patches can be found [here](https://huggingface.co/spaces/mayrajeo/maskrcnn-deadwood)\n",
    "\n",
    "Patch-level data are non-overlapping 512x512 pixel tiles extracted from larger virtual plots. The results presented here are the run with test-time augmentation.\n",
    "\n",
    "Scene-level data are the full virtual plots extracted from the full images. For Hiidenportti, the virtual plot sizes vary between 2560x2560px and 8192x4864px. These patches contain also non-annotated buffer areas in order to extract the complete annotated area. For Sudenpesänkangas, all 71 scenes are 100x100 meters (2063x2062) pixels, and during inference they are extracted from the full mosaic with enough buffer to cover the full area. The results presented here are run for 512x512 pixel tiles with 256 px overlap, with both edge filtering and mask merging described in the workflow."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6539d07a-e443-42ba-926e-598638c6396f",
   "metadata": {},
   "source": [
    "## Hiidenportti test set"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8551684f-3ae7-4172-ba69-33f965396e1a",
   "metadata": {},
   "source": [
    "Hiidenportti test set contains 241 non-overlapping 512x512 pixel image patches, extracted from 5 scenes that cover 11 circular field plots."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e62d486a-c05e-44b2-8f7a-2592def26f99",
   "metadata": {},
   "source": [
    "|Model|Patch AP50|Patch AP|Patch AP-groundwood|Patch AP-uprightwood|Scene AP50|Scene AP|Scene AP-groundwood|Scene AP-uprightwood|\n",
    "|----|-----------|--------|-------------------|--------------------|-----------|--------|-------------------|--------------------|\n",
    "|mask_rcnn_R_50_FPN_3x|0.654|0.339|0.316|0.363|0.640|0.315|0.235|0.396|\n",
    "|mask_rcnn_R_101_FPN_3x|**0.704**|**0.366**|0.326|**0.406**|**0.683**|**0.341**|0.246|**0.436**|\n",
    "|mask_rcnn_X_101_32x8d_FPN_3x|0.679|0.355|**0.333**|0.377|0.661|0.333|**0.255**|0.412|\n",
    "|cascade_mask_rcnn_R_50_FPN_3x|0.652|0.345|0.306|0.384|0.623|0.317|0.223|0.411|"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd8939f3-2e6f-4a1e-a701-be1d35894535",
   "metadata": {},
   "source": [
    "## Sudenpesänkangas dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6825d91-2708-4b79-b5b3-71d71a4bf305",
   "metadata": {},
   "source": [
    "Sudenpesänkangas dataset contains 798 on-overlapping 512x512 pixel image patches, extracted from 71 scenes."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "277dc82a-9621-468b-ab36-4d7c1a9e8aac",
   "metadata": {},
   "source": [
    "|Model|Patch AP50|Patch AP|Patch AP-groundwood|Patch AP-uprightwood|Scene AP50|Scene AP|Scene AP-groundwood|Scene AP-uprightwood|\n",
    "|----|-----------|--------|-------------------|--------------------|-----------|--------|-------------------|--------------------|\n",
    "|mask_rcnn_R_50_FPN_3x|0.486|0.237|0.175|0.299|0.474|0.221|0.152|0.290|\n",
    "|mask_rcnn_R_101_FPN_3x|**0.519**|**0.252**|**0.183**|0.321|**0.511**|**0.236**|**0.160**|**0.311**|\n",
    "|mask_rcnn_X_101_32x8d_FPN_3x|0.502|0.245|0.182|0.307|0.494|0.232|0.159|0.305|\n",
    "|cascade_mask_rcnn_R_50_FPN_3x|0.497|0.248|0.172|**0.323**|0.473|0.225|0.148|0.302|"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "855aaf89",
   "metadata": {},
   "source": [
    "# Running the models"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "432ded44-2aca-42f8-b576-a3a509c70b35",
   "metadata": {},
   "source": [
    "## Running models for image patches"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4dc5c9b1-5fa8-45d2-8cd1-c3a47f8a851a",
   "metadata": {},
   "source": [
    "For individual image patches, the models are fairly straightforward to run. \n",
    "\n",
    "```python\n",
    "from detectron2.engine import DefaultPredictor\n",
    "from detectron2.config import get_cfg\n",
    "from detectron2.data import build_detection_test_loader\n",
    "import cv2\n",
    "\n",
    "cfg = get_cfg()\n",
    "cfg.merge_from_file(<path_to_model_config>)\n",
    "cfg.OUTPUT_DIR = '<path_to_output>'\n",
    "cfg.MODEL.WEIGHTS = '<path_to_weights>'\n",
    "cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5 # score threshold for detections\n",
    "predictor = DefaultPredictor(cfg)\n",
    "\n",
    "img = cv2.imread('<path_to_image_patch>')\n",
    "outputs = predictor(image)\n",
    "```\n",
    "\n",
    "More examples are shown on [Patch level results](4_patch_level_results.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24b6c1e9-39d1-4030-92a6-1fcaaadc93ac",
   "metadata": {},
   "source": [
    "## Running models for larger scenes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22fdfc85-8462-48a0-98c5-f28c4fd55eba",
   "metadata": {},
   "source": [
    "Running on larger scenes requires the following steps:\n",
    "\n",
    "1. Tiling the scenes into smaller image patches, optionally with overlap\n",
    "2. Running the model on these smaller patches\n",
    "3. Gathering the predictions into a single GIS data file\n",
    "4. Optionally post-processing the results\n",
    "\n",
    "[drone_detector](https://jaeeolma.github.io/drone_detector) package has helpers for this:\n",
    "\n",
    "```python\n",
    "from drone_detector.engines.detectron2.predict import predict_instance_masks\n",
    "\n",
    "predict_instance_masks(path_to_model_config='<path_to_model_config>', # model config file\n",
    "                       path_to_image='<path_to_image>', # which image to process\n",
    "                       outfile='<name_for_predictions>.geojson', # where to save the results\n",
    "                       processing_dir='temp', # directory for temporary files, deleted afterwards. Default: temp\n",
    "                       tile_size=512, # image patch size in pixels, square patches. Default: 400\n",
    "                       tile_overlap=256, # overlap between tiles. Default: 100\n",
    "                       smooth_preds=False, # not yet implemented, at some points runs dilation+erosion to smooth polygons. Default: False\n",
    "                       coco_set='<path_to_coco>', # the coco set the model was trained on to infer the class names. If empty, defaults to dummy categories. Default: None\n",
    "                       postproc_results=True # whether to discard masks in the edge regions of patches Default: False\n",
    "                      )\n",
    "```\n",
    "\n",
    "Also, after installing the package, `predict_instance_masks_detectron2` can be used as CLI command with identical syntax.\n",
    "\n",
    "When provided models default to dummy classes, **1** is *standing deadwood* and **2** is *fallen deadwood*.\n",
    "\n",
    "More examples are shown on [Scene level results](5_scene_level_results.ipynb)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}