Validation-first EEG pipelines: how synthetic benchmarks mislead, and how pre-registration catches it

Automated EEG analysis pipelines are usually developed and evaluated on the samesynthetic or public data, and the outcome measures are chosen after the results areknown. We built an open, deterministic scalp-EEG pipeline (quality control, zero-phasefiltering, aperiodic/periodic spectral parameterisation, debiased weighted phase-lagindex, an unsupervised rhythmic-event detector and a citation-checked plain-languagesummary) under a validation-first protocol: six hypotheses with numeric pass thresholdswere committed to version control before the code or the data they concern existed,every method was tested against synthetic ground truth, and the test suite and theexplanation checker were themselves mutation-tested. Four of the six hypotheses wererejected. The protocol’s value lay in how they failed. A synthetic detection benchmark onwhich the detector passed (sensitivity 0.955 at 0.69 false detections per hour) hid fourfailure modes that only appeared on real recordings or on deliberately harder syntheticnegatives: a rhythm gate fooled by the 1/f background (the very confound one hypothesiswas written to test), harmonics of spike-wave discharges, normal alpha and spindlebursts, and blink trains merged into “rhythms”. On a harder benchmark the detector fellto 0.847 [95 % CI 0.803-0.882] at 1.68 false detections per hour, and on a pre-registeredheld-out subject of the CHB-MIT database it found 4 of 7 seizures at 3.3 false detectionsper hour. The pre-registered estimator thresholds also exposed the small-sample bias ofplain wPLI (95th percentile 0.19 under pure volume conduction at two minutes), and amechanical claim checker rejected all 2157 live mutated explanations once a rounding-tiedefect found on its first run was fixed. We argue that, for methods papers, the benchmarkitself is a researcher degree of freedom, and we give a short, checkable protocol toconstrain it. Code, data-retrieval scripts with checksums and all result files:https://doi.org/10.5281/zenodo.23447346.