Multi-Modal Pneumonia Detection

Pneumonia is a major global cause of morbidity and mortality, making early detection and accurate diagnosis crucial for effective treatment. Traditional diagnostic methods, such as radiological interpretation and clinical assessments, can be time-consuming and subject to observer variability. Deep learning models have revolutionized medical diagnostics by providing automated, reliable, and precise classification tools.This paper introduces a multi-modal pneumonia detection system that integrates EfficientNetB0 for Chest X-ray analysis and YAMNet for respiratory sound classification to enhance diagnostic accuracy. The dataset comprises 8,000+ annotated Chest X-rays and 100 respiratory sound samples from publicly available sources. The Chest X-ray images undergo preprocessing techniques such as grayscale conversion, normalization, median filtering, Contrast Limited Adaptive Histogram Equalization (CLAHE), and data augmentation, while audio samples are processed into log-mel spectrograms for deep learning-based classification. The proposed system fuses the predictions from both imaging and audio modalities, aiming to improve diagnostic reliability, particularly in resource-limited settings. Performance evaluation is conducted using F1 score, confusion matrix analysis, and accuracy metrics. Experimental results demonstrate that combining Chest X-ray and audio-based diagnosis reduces false negatives and increases confidence in pneumonia detection. Future work aims to collect paired Chest X-ray and respiratory audio data from the same patients to further enhance model precision