AI-driven multimodal imaging fusion using swin transformer and optimized tensor fusion networks for pneumonia detection

Pneumonia is a severe respiratory infection that significantly contributes to global morbidity and mortality, particularly among children and the elderly. Early and accurate detection of pneumonia is crucial for timely intervention and effective treatment, reducing the risk of severe complications. Traditional diagnostic methods, such as radiographic examination of chest X-rays (CXRs) and computed tomography (CT) scans, rely on the expertise of radiologists, which can lead to subjectivity and variability in diagnosis. The rapid advancement of deep learning in medical imaging has opened new possibilities for automated pneumonia detection, enabling faster, more accurate, and scalable diagnostic solutions. An important health challenge is Pneumonia and it is the timely and more accurate diagnosis. The multiple imaging modalities including CT-scans, X-rays and other diagnostic data integrated with the proposed AI driven framework thereby the pneumonia diagnosis robustness and accuracy enhanced. For effective multi-modal fusion, an effective deep learning models employed and from various imaging sources, the complementary information leveraged. To enhance the patient’s diagnostic results, an interpretable, timely and accurate detection model required. Initially perform pre-processing to neglect the artifact and noise of both CT-scans and X-rays image data. From these data, the relevant features extracted using Swin Transformers (ST). After that, the complex interactions among the imaging modalities are fused by employing Optimized Tensor Fusion Networks (OTFN). The Gradient-weighted Class Activation Mapping with Bayesian Neural Networks (Grad-CAM with BNN) is proposed for pre-emptive prediction of pneumonia disease. The risk assessment evaluated using the risk scoring system that provides a real-time alerts depending upon the predictive outputs thereby offering an early invention of pneumonia disease. Use python platform for implementation followed by the proposed work performance is evaluated using state-of-art studies.