Learning Anatomy-Disease Entangled Representation

Human experts demonstrate proficiency not only in disentangling anatomical structures from disease conditions but also in intertwining anatomical and disease information to accurately diagnose a variety of disorders. However, deep learning models, despite their prowess in acquiring intricate representation, often struggle to learn representation where distinct semantic aspects of the data (both anatomy and pathology) are entangled, particularly in medical images, which present a rich array of anatomical structures and potential pathological conditions. We envision that a deep model, when trained to comprehend medical images akin to human perception, would offer powerful representation with higher generalizability, robustness, and interpretability. To realize this vision, we have developed LeADER, a framework for learning anatomy-disease entangled representation from medical images. As a proof of concept, we have trained LeADER on ≈IM chest radiographs gatheredfrom 10 public datasets. Experimental results across 11 medical tasks, compared to 8 baselines in zero-shot, linear probing, limited data regimes, and full fine-tuning settings, demonstrate LeADER's superior performance over the Google CXR Foundation Model, large-scale medical models, and fully/self-supervised baselines across diverse downstream tasks. This enhanced performance is attributed to the significance of entangling anatomy-specific and disease-specific representations via our framework, which enables the simultaneous acquisition of both anatomical and disease knowledge, yet overlooked in existing supervised/self-supervised learning methods. All code and models are available at GitHub.com/JLiangLab/LeADER.