Automated assessment of Crohn’s disease activity using a deep learning model with multimodal data

Abstract Background Accurate prediction of disease activity levels in Crohn’s disease (CD) is essential for understanding disease progression and guiding clinical management. However, this remains challenging due to heterogeneous clinical manifestations and complex underlying mechanisms. This study aimed to develop a multimodal deep learning framework that integrates routine blood test data with endoscopic images to improve the prediction of disease activity levels. Methods This retrospective study included 397 endoscopic examinations (397 videos) and their corresponding blood test records from 226 patients with confirmed Crohn’s disease (CD). A total of 3,234 endoscopic images were extracted from these videos for model development. Based on these multimodal data, a deep learning model was designed to jointly learn representations from blood biomarkers and endoscopic imaging features. The model was trained and evaluated using five-fold cross-validation, and its performance was compared with models based on single data modalities. Results The proposed model achieved an area under the receiver operating characteristic curve of 0.7843, an accuracy of 0.7179, a recall of 0.6245, and an F1 score of 0.6523. Compared with models using only blood test data or only endoscopic images, the multimodal approach improved the area under the curve by 2.65% and 13.70%, respectively, and improved accuracy by 16.88% and 13.60%, respectively. Conclusions The multimodal framework effectively distinguishes among remission, mild, moderate, and severe disease activity levels in CD. These findings demonstrate the potential of artificial intelligence-based multimodal learning to support more accurate and clinically useful disease activity assessment. Clinical trial number Not applicable.