Noise-Aware Evaluation of Object Detectors

Supervised object detection requires annotated datasets for training and evaluation purposes. However, human annotation of large datasets is error-prone, and frequent mistakes are erroneous labels, missing objects, and impre-cise bounding boxes. The main goals of this work are to quantify the extent of annotation noise in terms of corner-wise discrepancies, assess how it impacts evaluation met-rics for object detection, and propose noise-aware alter-natives that serve as upper and lower bounds for a base-line metric. We focus our analysis on the Microsoft COCO dataset and re-evaluate several state-of-the-art object de-tectors using the proposed metrics. We show that the Average Precision (AP) metric might be considerably over or under-estimated, particularly for small objects and restrictive IoU acceptance thresholds. Our code is available at https://github.com/Artcsl/Error-Aware.