Semi-Supervised Multimodal Multi-Instance Learning for Aortic Stenosis Diagnosis

Automated interpretation of ultrasound imaging of the heart (echocardiograms) could improve the detection and treatment of aortic stenosis (AS), a deadly heart disease. However, existing deep learning pipelines for assessing AS from echocardiograms have two key limitations. First, most methods rely on limited 2D cineloops, thereby ignoring widely available Spectral Doppler imaging that contains important complementary information about pressure gradients and blood flow abnormalities associated with AS. Second, obtaining labeled data is difficult. There are often far more unlabeled echocardiogram recordings available, but these remain underutilized by existing methods. To overcome these limitations, we introduce Semi-supervised Multimodal Multiple-Instance Learning (SMMIL), a new deep learning framework for automatic interpretation for structural heart diseases like AS. During training, SMMIL can combine a smaller labeled set and an abundant unlabeled set of both 2D and Doppler modalities to improve its classifier. When deployed, SMMIL can combine information from all available images to produce an accurate study-level diagnosis of this life-threatening condition. Experiments demonstrate that SMMIL outperforms recent alternatives, including two medical foundation models.