Guidelines for the Selection, Calibration, and Evaluation of Post-hoc OOD Detectors in High-Dimensional Representations Generated by CNN and ViT Models in Image Recognition

Highlights • The performance of post-hoc OOD detectors is deep model type-specific. • Guidelines on the recommended post-hoc OOD detector for different CNN or ViT architectures are formulated. • The Mahalanobis OOD detector with pool covariance matrix should be discouraged; diagonal covariance matrices yield improved performance. • Evaluating OOD 2detectors using per-class AUROC reveals OOD generalization risks in deep model deployment. • Modern CNN and ViT models include ID classes with surprisingly poor OOD generalization, despite high overall AUROC in OOD detection benchmarks. Detecting out-of-distribution (OOD) examples is essential for using machine learning in safety-critical applications. Although many OOD methods have been proposed, the literature does not provide conclusive recommendations on which OOD detection method should be used in practice, as the performance of OOD detectors is benchmark-specific. We comprehensively analyzed the performance of parametric and nonparametric post-hoc OOD detectors in representation spaces generated by several lines of CNNs and ViT architectures. On this basis, we pointed out each deep model’s preferred post-hoc OOD detector(s), which consistently outperformed other detectors regardless of the OOD benchmark dataset. We analyzed the properties of the high-dimensional representations generated by different deep networks. We challenge the assumptions underlying the prominent Mahalanobis distance-based OOD detector with the pooled covariance matrix. We analyzed the impact of the simplifying assumptions (such as common vs. diagonal covariance matrices). We showed that the performance of the Mahalanobis method improves when more realistic simplifications are made, given the actual characteristics of the data. We also quantified the impact on OOD detection of the unreliable probability density estimation in high-dimensional data - as the Mahalanobis method relies on the MVN model fitted to sparse data. We show how this curse of dimensionality phenomenon affects the choice of the OOD detection threshold. Finally, we propose the extended metrics for evaluating the OOD detectors. Using these metrics, we show that many CNN and ViT models include classes with surprisingly poor OOD generalization despite high overall AUROC in OOD detection benchmarks.