Why Good Detectors Can Make Bad Trackers: mAP Worse than Random Chance at Model Selection

Neural network models have emerged over the last decade as the standard for perception in many robotic systems. While powerful, these models are opaque and practitioners often rely on empirical evaluation using standard metrics and held-out test sets to understand their performance. However, such model-level testing ignores how the outputs of perception models will be used in downstream components within a robotic system. In this work, we create a new family of object detection metrics that can accurately evaluate performance when detectors are used for object tracking in a robotic system. We show that mean Average Precision (mAP), arguably the most common metric for evaluating object detectors, does not strongly correlate with tracking performance. We propose an alternative family of metrics, the Match’n Metric, that makes few, simple assumptions about downstream tracking algorithms and does not require the implementation of a tracker to compute. We show empirically that across a number of object detection models and tracking algorithms that Match’n Metric more accurately correlates with tracking performance than mAP. Equipped with this metric, practitioners can better evaluate object detectors for their ability to be used in robotic systems that require object tracking.