DetectionMetrics

Project webpage here

⚠️ DetectionMetrics v1 website referenced in our Sensors paper is still available here

DetectionMetrics is a family of toolkits designed to unify and streamline the evaluation of perception models across different frameworks and datasets. Looking for our published DetectionMetrics v1? Check out all the relevant links below.

Now, we're excited to introduce DetectionMetrics v2! While retaining the flexibility of our previous release, DetectionMetrics has been redesigned with an expanded focus on image segmentation, with plans to extend support to object detection and LiDAR applications. As we move forward, v2 will be the actively maintained version, featuring continued updates and enhancements to keep pace with evolving AI and computer vision technologies.

💻 Code	🔧 Installation	🧩 Compatibility	📖 Docs

What's supported in DetectionMetrics

Task	Modality	Datasets	Framework
Segmentation	Image	Rellis3D, GOOSE, custom GAIA format	PyTorch, Tensorflow
	LiDAR	Coming soon	Coming soon
Object detection	Image	Check DetectionMetrics v1	Check DetectionMetrics v1

More details about the specific metrics and input/output formats required fow each framework are provided in the Compatibility section in our webpage.

Installation

In the near future, DetectionMetrics is planned to be deployed in PyPI. In the meantime, you can clone our repo and build the package locally using either venv or Poetry.

Using venv

Create your virtual environment:

mkdir .venv
python3 -m venv .venv

Activate your environment and install as pip package:

source .venv/bin/activate
pip install -e .

Using Poetry

Install Poetry (if not done before):

python3 -m pip install --user pipx
pipx install poetry

Install dependencies and activate poetry environment (you can get out of the Poetry shell by running exit):

poetry install
poetry shell

Common

Install your deep learning framework of preference in your environment. We have tested:

• CUDA Version: 12.6
• torch==2.4.1
• torchvision==0.19.1
• tensorflow[and-cuda]==2.17.1

And it's done! You can check the examples directory for inspiration and run some of the scripts provided either by activating the created environment using poetry shell or directly running poetry run python examples/<some_python_script.py>.

DetectionMetrics v1

Our previous release, DetectionMetrics v1, introduced a versatile suite focused on object detection, supporting cross-framework evaluation and analysis. Cite our work if you use it in your research!

💻 Code	📖 Docs	🐋 Docker	📰 Paper

Cite our work

@article{PaniegoOSAssessment2022,
  author = {Paniego, Sergio and Sharma, Vinay and Cañas, José María},
  title = {Open Source Assessment of Deep Learning Visual Object Detection},
  journal = {Sensors},
  volume = {22},
  year = {2022},
  number = {12},
  article-number = {4575},
  url = {https://www.mdpi.com/1424-8220/22/12/4575},
  pubmedid = {35746357},
  issn = {1424-8220},
  doi = {10.3390/s22124575},
}