Transformer Architectures for Respiratory Sound Analysis and Multimodal Diagnosis

Abstract Breath sound analysis is one of the diagnostic approaches for various lung diseases. However, traditional auscultation is subjective and dependent on the physician’s experience. Recent studies have demonstrated that computer-assisted methods analyzing these sounds can significantly improve the accuracy of diagnosis of the patient’s condition. However, the absence of a common basis for fair comparison, limited use of patient metadata, and lack of interpretability make these approaches difficult to adopt in clinical practice. In the paper, we propose to utilize the benefits of the recent state-of-the-art sound classification models and a multimodal approach for respiratory disease screening and diagnostics. To achieve this goal, we use the Audio Spectrogram Transformer (AST) model and a vision-language model (VLM) Moondream2. We compare them with our previous work on the DenseNet201 Convolutional Neural Network (CNN). The proposed methodology allows a fair comparison of different computer-assisted methods for respiratory disease screening and diagnostics. The training and evaluation of the models have been conducted using the same dataset containing respiratory sound recordings and patient information. The obtained results demonstrate that these transformer-based models can serve as a basis for the development of computer-assisted methods for asthma diagnosis, as well as for continuous and remote monitoring of patient condition. In addition, our approach enables interpretability of the model performance. The achieved performance is much better than the best results of the physician’s auscultation.