Out-of-Distribution Detection in Medical Image Segmentation with $$\beta $$-VAE and Likelihood Regret

The performance of deep learning models can be compromised in the presence of Out-of-Distribution (OOD) data, i.e., data which deviates from the training distribution, the IN-distribution. This work examines OOD detection for medical image segmentation, introducing a new method that simultaneously detects which image samples are OOD, but also finds if the segmentation masks are OOD. The latter problem may arise if there are corruptions or discrepancies in the ground truth annotation masks, a likely scenario in real-world medical imaging applications, which has not received research attention to date. Most works focus on sample-wise OOD detection, finding samples that do not follow the IN-distribution, or pixel-wise OOD detection, which finds pixels that do not follow the IN distribution (anomaly detection), but do not examine if the segmentation mask itself is OOD. We propose a new model for simultaneous OOD detection in image samples and annotations, based on a combination of <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mi>β</mml:mi></mml:math> -VAE for the partial disentanglement of the data latent space, and U-Net for segmentation. We introduce the use of Likelihood Regret to compute OOD scores for both input reconstruction and the segmentation, for a more accurate assessment of the goodness-of-fit of distributions, as likelihood alone has been shown to assign higher values to OOD samples, resulting in misleading OOD detection. Experimental results demonstrate the proposed method's efficiency at discriminating OOD data for 3D medical image segmentation in a systematic manner, showing excellent performance for large distribution drifts and competitive performance for very small distribution drifts.