Assessing the robustness of evaluation metrics for synthetic ECG signal quality

The evaluation of synthetic biomedical data remains inconsistent across studies, limiting comparability and reproducibility. This work examines the robustness of evaluation metrics used for synthetic data assessment, using electrocardiograms (ECGs) as a case study. Five generative models, SSSD-ECG, WaveGAN*, Pulse2Pulse, cNVAE-ECG and YData's proprietary model, were assessed through metrics covering fidelity, diversity, privacy, and utility dimensions. Controlled perturbations were applied to synthetic ECGs across magnitude, time, and frequency domains to examine whether metrics remained stable under clinically irrelevant changes while responding to progressive degradations in waveform morphology. Results revealed distinct behaviors across quality dimensions. Within fidelity, density consistently captured structural distortions. For diversity, coverage provided a reliable assessment, decreasing as perturbations intensified. Privacy metrics increased with transformations, confirming the expected fidelity-privacy trade-off, whereas utility, measured by macro-AUROC, remained stable under moderate perturbations and declined only when diagnostic information was severely disrupted. Overall, findings indicate that metric robustness varies across quality dimensions and many of the commonly used metrics may fail to reflect meaningful signal degradations. The proposed perturbation analysis provides relevant insights, constituting a first step toward more reliable and interpretable evaluation pipelines for synthetic time-series data and underscores the need for continued research on robust evaluation metrics.