Multimodal contrastive learning for optimized integration of ECG signals and clinical reports for arrhythmia classification

Early and accurate detection of cardiac arrhythmias is essential for timely diagnosis and effective treatment. Traditional ECG-based classification models primarily rely on signal processing techniques, often neglecting the valuable contextual information contained in clinical notes. ‎This study introduces a novel multimodal contrastive learning framework that integrates ECG signals and clinical notes for arrhythmia clas-‎sification. This approach is not previously explored in ECG-based diagnostics. Unlike existing methods that process ECG and clinical text ‎separately, our framework employs contrastive learning for early feature alignment, ensuring meaningful representation of both modalities. ‎Additionally, we propose an alternative alignment strategy that enables ECG-text integration without requiring direct patient-level mapping, ‎addressing a major challenge in real-world healthcare applications. A neural network extracts temporal features from ECG signals, while a ‎BERT-based model processes textual clinical data. Contrastive learning is employed to align multimodal representations, improving the ‎model’s ability to differentiate between normal and abnormal arrhythmias. Experimental results demonstrate 97.3% accuracy and an AUC-‎ROC of 0.98, significantly outperforming unimodal approaches. The proposed method enhances model generalization, robustness, and ‎interpretability, making it suitable for real-world clinical applications. This study contributes to the development of AI-driven diagnostic ‎tools that can integrate multimodal patient data, improving the reliability of automated arrhythmia detection in healthcare settings‎.