Heartbeat: a multimodal dataset of fetal echocardiography and clinical metadata for early detection of congenital heart disease

Background: Congenital heart diseases (CHDs) remain the leading cause of infant mortality attributable to birth defects. Although artificial intelligence has demonstrated promise for automated CHD detection from prenatal ultrasound imaging, progress is constrained by the limited availability of large, high-quality datasets and the absence of multimodal resources integrating imaging with clinically relevant patient metadata. Methods: This study introduces Heartbeat, a multicenter, anonymized multimodal dataset comprising fetal cardiac ultrasound images and associated clinical metadata collected between 2019 and 2023. Standard echocardiographic views, including the three-vessel trachea, four-chamber, left ventricular outflow tract, and right ventricular outflow tract planes, were acquired according to established protocols. Multiple deep learning architectures, including convolutional neural networks and a transformer-based baseline, were trained and evaluated. Heart-ViT, a multimodal transformer-based model, was developed to integrate imaging features with patient-specific clinical metadata via adaptive layer normalization. Results: ), Heart-ViT achieved a sensitivity of 79.17%, specificity of 93.18%, PPV of 56.95%, AUROC of 88.56%, and an F1-score of 65.63%, outperforming multiple image-only convolutional neural network architectures and baseline models. Multimodal integration yielded a 13.22-point improvement in F1-score relative to the imaging-only transformer baseline. Conclusion: Heartbeat provides a clinically relevant multimodal resource for prenatal CHD research that reflects real-world imaging variability and class imbalance. Integrating ultrasound imaging with patient-specific clinical metadata significantly enhances CHD detection compared with image-only approaches. These findings establish a reproducible benchmark and support the development of AI-assisted tools for early prenatal screening. The dataset and analytical framework will be made publicly available to support further research in prenatal cardiology and the early detection of CHDs.