Deep Analysis of YOLOv5’s robustness properties

[mrojascarulla]

We’ve recently evaluated the robustness of several object detection models, including YOLO, and published our findings in this blog post: https://www.lakera.ai/insights/pre-trained-poor-generalization.

We found that YOLO models are more robust than transformer based models, yet they remain brittle, even against augmentations that were used during training. This is relevant for building systems for production which need to be robust to unaccounted input variations.

I am interested in understanding how one should go about choosing a base model to finetune, as some of these robustness properties may be inherited by systems built on top.

Here are a few questions I would love to get your input on:

• How do you usually select a base model? Can you share what has worked for you?
• How could we go about making these base YOLO models more robust, and thus more likely to generalize?

[glenn-jocher]

@mrojascarulla thanks for the insights! We've always strived to make YOLOv5 a real-world AI and your blog post reflects these results well :)

EDIT: To answer your questions, we develop models based on COCO results, which are somewhat applicable but not as broadly applicable as say a basket of real-world datasets on common use cases, i.e. manufacturing, self-driving, healtchare. If we had more resources we'd probably design for the Objects365 dataset, which is an order of magnitude larger and encompasses 4x more classes than COCO.

[glenn-jocher]

@mrojascarulla not to mention we officially support YOLOv5 export to almost every format for easy and guaranteed deployment simplicity, which is not the case for the vast majority of other architectures.

Formats

YOLOv5 inference is officially supported in 11 formats:

💡 ProTip: Export to ONNX or OpenVINO for up to 3x CPU speedup. See CPU Benchmarks.
💡 ProTip: Export to TensorRT for up to 5x GPU speedup. See GPU Benchmarks.

Format	export.py --include	Model
PyTorch	-	yolov5s.pt
TorchScript	torchscript	yolov5s.torchscript
ONNX	onnx	yolov5s.onnx
OpenVINO	openvino	yolov5s_openvino_model/
TensorRT	engine	yolov5s.engine
CoreML	coreml	yolov5s.mlmodel
TensorFlow SavedModel	saved_model	yolov5s_saved_model/
TensorFlow GraphDef	pb	yolov5s.pb
TensorFlow Lite	tflite	yolov5s.tflite
TensorFlow Edge TPU	edgetpu	yolov5s_edgetpu.tflite
TensorFlow.js	tfjs	yolov5s_web_model/
PaddlePaddle	paddle	yolov5s_paddle_model/

[AyushExel]

@mrojascarulla Very informative blog! Thanks for sharing

[mrojascarulla]

Thanks for the reply @glenn-jocher! Looking at a larger dataset makes sense, have you thought of also adding augmentations more aggressively during training and testing for robustness properties on the final models? The experiments suggest that this would already help in making the base models more robust.

I'm also curious whether you have some feedback from people building production systems on top of YOLO? Do you know how they go about making their systems more robust?

[glenn-jocher]

@mrojascarulla yes often a validation set doesn't completely encompass the variety found in real-world deployments, so often real-world results will benefit from more aggressive augmentation despite the fact that this may reduce validation mAP.

This is hard to quantify though of course, so the safest strategy is to ensure that your validation set is a representative subset of your deployment space. If not your trained model may not generalize well to the real world, which is one of the common failure modes we observe in the slide below:

[MBichurin]

What a great discussion
I always wondered why YOLOv5 uses one single augmentation during training. It's sad there is no paper on the model. @glenn-jocher can you tell me if you experimented with different augmentation setups during training? And if so, can you give some insights on those experiments?
Thank you!

[glenn-jocher]

@MBichurin yes, we experimented with a lot of augmentation setups, all of them with the purpose of improving COCO training from scratch. This is not the same as the default use case though, which is training smaller datasets starting from COCO-pretrained models, so a better hyperparameter search for the best real-world applicability would be to apply to a basket of smaller datasets.

We did find that smaller models overfit later and thus benefit less from augmentation, while larger models overfit earlier and benefit from stronger augmentation. This is why we have 3 levels of augmentation hyp files in https://github.com/ultralytics/yolov5/tree/master/data/hyps