Deep transfer learning-based approaches with multimodal fusion to improve the prediction of knee osteoarthritis progression: data from the OAI and MOST cohorts

Abstract Effective diagnosis and management of knee osteoarthritis (KOA) increasingly rely on integrating diverse data sources, including imaging and clinical information. This study aimed to evaluate the use of multimodal deep learning models assisted by transfer learning (TL) to enhance the prediction of KOA progression. Deep learning models (ResNet-34 and DenseNet-201) were employed for magnetic resonance imaging (MRI) and X-ray feature extraction. A set of single-modality, dual-modality, and multimodal configurations incorporating MRI, X-ray, and radio-clinical data were evaluated using intermediate fusion. A total of 1714 patients were selected from the Osteoarthritis Initiative (OAI) and Multicenter Osteoarthritis Study (MOST) cohorts. TL strategies included one-step transfer using ImageNet pretrained weights and two-step transfer initially pretraining on one KOA cohort (OAI or MOST) and then fine-tuning and validating on an external KOA cohort (MOST or OAI). Results demonstrated that ImageNet-based TL significantly improved the predictive performance of X-ray models, with AUC scores exceeding 0.7 in both the OAI and MOST cohorts. For MRI-based models, the lack of large-scale pretrained weights limited the performance benefit of TL. Multimodal configurations offered no clear advantage over unimodal or dual-modality models. Without TL, DenseNet outperformed ResNet. These findings underscore the potential of transfer learning in predicting radiographic KOA progression.