Self-supervised Prototype Learning for Spatio-Temporal Enhanced Ultrasound-based Prostate Cancer Detection

Prostate cancer detection in ultrasound data presents significant challenges due to the highly heterogeneous nature of the cancer and its appearance in ultrasound images. In this work, we introduce a novel multi-modal framework that integrates spatial information in ultrasound images with temporal time-series signals across a sequence of ultrasound images for robust prostate cancer detection. Our framework employs a clustering-based self-supervised learning approach, which allows the model to learn shared, complementary representations from combined spatio-temporal data. This approach not only captures the unique properties of spatial and temporal information, but also enhances the model's ability to handle noisy labels due to imperfect alignment between imaging data and histopathology reports, resulting in improved classification performance. Our method achieves a 7% improvement in AUC and a 10% increase in balanced accuracy compared to models relying on spatial or temporal ultrasound data alone. Our code and visualizations can be found in https://github.com/DeepRCL/Multi-SWAV.