Multimodal model for pneumonia detection based on enhanced stacking MOE

The use of artificial intelligence technology for pneumonia detection has always been an important topic in medical imaging, and integrating and processing various medical data sources to enhance the early diagnosis and identification of pneumonia has become a key challenge in the healthcare field. This study proposes a novel pneumonia prediction model with a hybrid architecture, which integrates the most advanced large language models and a stacking MOE architecture. The model combines the pre-trained ResNet-50 on ImageNet for effective image feature extraction, while using BERT to extract and analyze textual data from patient self-reports. This study uses a small-scale dataset to test pneumonia prediction based on X-ray images and patient self-reported text. The experimental results show that the hybrid model outperforms the single-modality models in all evaluation metrics. Additionally, this study conducted an ablation test on the stacking MOE and LLMs components, demonstrating the effectiveness of each component. The research results provide a feasible approach for the potential of multimodal artificial intelligence models in accurately detecting pneumonia and contribute to the development of flexible clinical diagnostic tools.