metadata.json

{
    "title": "Thermal Bridges on Building Rooftops Detection (TBBRDet)",
    "summary": "Identify thermal bridges on building rooftops via object detection / instance segmentation models",
    "description": [
        "The application adapts object detection / instance segmentation models from the ",
        "[MMDetection Toolbox](https://github.com/open-mmlab/mmdetection/tree/v2.21.0) ",
        "to work with combined RGB and thermal imagery to identify thermal bridges (weak points) on building rooftops.",
        "",
        "<video controls><source src='https://user-images.githubusercontent.com/12682506/177049125-36658a55-e07e-4355-8fbe-10d422cf7246.mp4' type='video/mp4'>Your browser does not support the video tag.</video>\n",
        "",
        "**Training the deep learning model(s)**\n",
        "The service is based on the [TBBRDet code](https://github.com/Helmholtz-AI-Energy/TBBRDet)" ,
        "which forms the basis of the paper [Investigating deep learning approaches to building rooftop thermal bridge ",
        "detection from aerial images](https://doi.org/10.1016/j.autcon.2022.104690) [1]. ",
        " It describes an ablation study with the model architectures:\n",
        "* MaskRCNN ResNet-50 backbone",
        "* MaskRCNN Swin-T Transformer backbone",
        "* TridentNet*",
        "* Feature Selective Anchor-Free (FSAF)*",
        "",
        "The study varies model training from scratch and with [COCO](https://cocodataset.org/) pretrained weights, ",
        "as well as using height maps (5-channel inputs) vs standard RGBT (4-channel inputs).\n",
        "The Swin-T model (with 5-channel inputs and pretrained weights) achieved the highest average recall scores, ",
        "so this application focuses on that architecture. The pretrained weights for the Swin model can be downloaded ",
        "[here](https://download.openmmlab.com/mmdetection/v2.0/swin/mask_rcnn_swin-t-p4-w7_fpn_fp16_ms-crop-3x_coco/mask_rcnn_swin-t-p4-w7_fpn_fp16_ms-crop-3x_coco_20210908_165006-90a4008c.pth)",
        " and are also included in the docker image.",
        "",
        "Training via this application automatically creates a timestamp folder in a predefined model structure",
        " within the tbbrdet_api/models/ folder. You can check the status of any training by looking at the ",
        "automatically generated .log.json file in the associated timestamp folder.",
        "",
        "**Data**\n",
        "The dataset that forms the basis of model training is the [Thermal Bridges on Building Rooftops (TBBRv2)](https://doi.org/10.5281/zenodo.6517768).",
        "A simple, easily downloadable version can be found [on zenodo](https://zenodo.org/records/4767772). ",
        "The dataset contains **917 images** with a total of **6895 annotations** of thermal bridges on rooftops:\n",
        "* train (723 images)",
        "* test (203 images)",
        "",
        "Due to the small number of images, no validation set is created and instead the data is merely split into train / test. ",
        "For use in this application, it is sufficient to simply download all the .tar.std and .json files from the listed link. ",
        "The train / test folder structure will be built automatically by provided the folder path containing these files.\n",
        "Alternatively, you can place your own extracted data (.npy and .json files) in your directory of choice, ",
        "although it is essential to keep the same folder structure as shown ",
        "[on zenodo](https://zenodo.org/records/4767772)",
        "",
        "**Inference**\n",
        "",
        "The docker image contains a pretrained Swin-T model for inference (tbbrdet_api/models/swin/coco/2023-12-07_130038) ",
        "that expects 5-channel numpy inputs (RGB, T, H) formatted in the same way as those in the ",
        "zenodo dataset. The inference results are automatically saved to the used model's timestamp folder.",
        "",
        "",
        "**References**\n",
		"[1] Mayer, Z., Kahn, J., Hou, Y. et al. Deep learning approaches to building rooftop thermal bridge detection from aerial images. ",
        " In Automation in Construction Vol. 146, p. 104690 (2023). Elsevier BV. https://doi.org/10.1016/j.autcon.2022.104690, ",
        "",
		"[2] Mayer, Z., Kahn, J., Götz, M. et al. Thermal Bridges on Building Rooftops. ",
        "In Sci Data 10, 268 (2023). https://doi.org/10.1038/s41597-023-02140-z"
    ],
    "keywords": [
        "docker",
        "api-v2",
        "vo.ai4os.eu",
        "PyTorch",
        "deep learning",
        "pre-trained",
        "trainable",
        "inference",
        "object detection",
        "instance segmentation",
        "multi-spectral data"
    ],
    "license": "BSD-3-Clause",
    "date_creation": "2023-02-07",
    "dataset_url": "https://zenodo.org/records/4767772",
    "sources": {
		"dockerfile_repo": "https://github.com/emvollmer/tbbrdet_api/docker",
		"docker_registry_repo": "deephdc/uc-emvollmer-deep-oc-tbbrdet_api",
		"code": "https://github.com/emvollmer/tbbrdet_api"
	},
    "continuous_integration": {
        "build_status_badge": "https://jenkins.services.ai4os.eu/buildStatus/icon?job=AI4OS-HUB-TEST%2Ftbbrdet_api%2Fmaster",
        "build_status_url": "https://jenkins.services.ai4os.eu/job/AI4OS-HUB-TEST/job/tbbrdet_api/job/master/"
    },
    "tosca": [
        {
            "title": "Mesos (CPU)",
            "url": "https://raw.githubusercontent.com/indigo-dc/tosca-templates/master/deep-oc/deep-oc-marathon-webdav.yml",
            "inputs": [
                "rclone_conf",
                "rclone_url",
                "rclone_vendor",
                "rclone_user",
                "rclone_pass"
            ]
        }
    ]
}