Unsupervised hybrid framework for anomaly detection (HAND): applied to screening mammogram

Medical out-of-distribution (OOD) detection poses great challenges due to the heterogeneity and unknown characteristics of medical data. Therefore, to train a novelty detector with only ID data available, learning high-quality “normality” features is the fundamental step to identify the OOD samples during inference. Unfortunately, it still lacks an effective way to identify the difference for several datasets from the same medical domain. The main challenge lies in the inaccessibility to external medical datasets given the privacy concerns around sharing personally identifiable information. Therefore, an efficient way of external dataset curation by identification of anomaly without sharing data is desired, particularly for AI model generalization. Our proposed novel HAND architecture, composed of CNN+transformer, discriminator, and gradient reversal, can be trained in an unsupervised way to detect both intra- and inter-class anomalies from mammogram screening exams, with the AUC score of 0.94 and 0.85 for internal and external evaluation, achieves the best performance among VAE-based, GAN-based, and CNN+transformer baselines.