A reproducible benchmark for ECG-to-text report generation: Leakage-safe evaluation, performance floors, and stress testing

Free-text report generation directly from ECG signals is an emerging task, but the field currently lacks a reproducible, leakage-safe evaluation protocol that makes results comparable across studies. To address this, we introduce both a benchmark and a reference implementation: the benchmark provides a leakage-safe evaluation protocol with concrete performance floors, and the reference implementation provides a tuned reference system that future ECG-to-text methods can be compared against. Our central contribution is the benchmark: an evaluation protocol on PTB-XL using the official, patient-exclusive splits, together with two architecture-agnostic statistical floors (a text-only lexical baseline and a class-conditional deterministic template baseline) and two model-dependent floors (a label-only neural baseline and a nearest-neighbor retrieval baseline) that quantify how much of a model’s score can be explained by report priors, rhythm-level shortcuts, and latent-space retrieval alone. We additionally include a random-split negative control that quantifies leakage-driven score inflation, and a stress test on a proprietary single-lead subcutaneous ECG dataset from implantable cardiac monitors, with analysis of site-level reporting bias. As the reference implementation, we adapt a standard encoder–decoder, image-captioning-style pipeline (ResNet encoder with LSTM or Transformer decoder) to signal-to-text on PTB-XL. Under this protocol, the reference implementation reaches METEOR 55.53% on PTB-XL, clearing all four floors; the primary prior model reported 24.51% under a leakage-prone random-split setup. We read these gains as improvements on this benchmark rather than evidence of clinically grounded morphological understanding: evaluation relies on lexical-overlap metrics over short, templated reports, and the margin over the strongest retrieval baseline is modest.