Multimodal Models for Comprehensive Cardiac Diagnostics via ECG Interpretation

The swift progression of Multimodal Large Language Models (MLLMs) has exhibited substantial potential within medical contexts, particularly in facilitating patient engagements that incorporate physiological signal data. Despite these advancements, prevailing MLLMs demonstrate deficiencies in the accurate diagnosis of cardiac ailments, notably in the integration of electrocardiogram (ECG) data interpretation with the formulation of detailed medical reports. This inadequacy primarily arises from the inherent complexity of ECG signals and the disconnect between textual and signal-based modalities. Additionally, these models frequently encounter instability when generating extended medical narratives, attributable to a lack of specialized domain knowledge pertinent to user queries. To mitigate these challenges, we introduce ECG-Chat, the first multitask MLLM meticulously designed for the generation of ECG-based medical reports, thereby supporting multimodal interactions grounded in cardiological expertise. Our architecture employs an innovative contrastive learning framework that synchronizes ECG waveform data with textual reports, ensuring accurate alignment between ECG characteristics and their corresponding medical descriptions. This approach also facilitates the development of a resilient ECG encoder capable of zero-shot report retrieval. Furthermore, ECG-Chat is equipped to autonomously generate comprehensive ECG analysis reports through an automated LaTeX pipeline. We have established a new benchmark for ECG report generation and evaluated ECG-Chat against several baseline models. The results affirm that ECG-Chat surpasses current models in tasks including classification, retrieval, multimodal dialogue, and medical report generation. Notably, our report template design has received extensive validation from healthcare professionals.