Evaluating your Re-ID Method for Robustness: Consistency, Domain-Shift and Corruption

Despite high performance on benchmarks, machine learning methods can make serious errors under real world conditions. Using the application of target re-identification (Re-ID) as an example, this paper challenges eight state of the art pedestrian/vehicle Re-ID methods to perform consistently and accurately, and presents an analysis of their robustness. Four real world datasets, Market1501, CUHK01, GRID and VRIC, are used to examine three aspects of robustness: robustness to application, robustness to corruption, and robustness to domain shift. Surprisingly, for most of the eight methods studied, we find that small domain shifts or weak corruption are enough to cut their performance in half. We aggregate the results of these experiments and rank the Re-ID methods' overall performance. Of the tested methods, LightMBN is a clear winner in terms of robustness.