Unsupervised Generative Approach for Anomaly Detection to Enhance the Quality of Unseen Medical Datasets

Out-of-distribution (OOD) and anomaly detection are critical for reducing noise and improving the generalization of AI models in breast cancer screening, especially when handling unseen private data. Given the challenge of limited prior knowledge about OOD samples in external unseen datasets, unsupervised generative learning is a preferable solution which trains the model to discern the normal characteristics of in-distribution (ID) data. The hypothesis is that during inference, the model aims to reconstruct ID samples accurately, while OOD samples exhibit poorer reconstruction due to their divergence from normality. In-spired by state-of-the-art (SOTA) hybrid architectures combining CNNs and transformers, we developed a novel back-bone - HAND (Hybrid ANomaly Detection), for detecting OOD from digital mammogram studies. To boost learning efficiency, we incorporated synthetic OOD samples and a parallel discriminator in the latent space to distinguish between ID and OOD samples. Gradient reversal to the OOD reconstruction loss penalizes the model for learning OOD reconstructions. An anomaly score is computed by weighting the reconstruction and the discriminator loss. On internal held-out test and external dataset, the proposed HAND model outperformed encoder-based and GAN-based baselines, and interestingly, it also outperformed the hybrid CNN+transformer baselines. Proposed HAND pipeline of-fers an automated and efficient computational solution for domain-specific quality checks in external screening mam-mograms, yielding actionable insights without direct exposure to private medical imaging data.