Generalization or mirage? Data leakage and reported performance in neonatal EEG seizure detection models: a systematic review

The reliability of deep learning models in neonatal seizure detection depends on rigorous experimental design and transparent reporting of performance. In this systematic review, 48 peer-reviewed studies and 145 distinct study observations of EEG-based neonatal seizure detection were examined to assess how data partitioning (validation) strategies, open data (code), and data leakage influence reported performance. We analyzed the extracted data using Fisher’s exact test, chi-square test, and partial least squares structural equation modeling (PLS-SEM). Of the 48 studies included, public datasets were used in 87.69% of papers, but only 22.9% reported code availability. While studies with low risk of data leakage are common, we found that 16.7% of papers or 31.7% of study observations exhibited a high risk of data leakage. The PLS-SEM results and chi-square test align well, supporting that rigorous data partitioning reduces leakage and that leakage adversely affects performance. Data Partitioning (Validation) strategy emerged as a strong and statistically significant predictor of data leakage (p < 0.001, R2 = 0.598), explaining 59.8% of its variance. Data leakage has a statistically significant effect on reported model performance (p = 0.005, R2 = 0.202). Still, they account for only about 20.2% of the variation in performance, indicating other factors might play a substantial role, such as dataset size, class balance, model architecture, and preprocessing. Overall, the evidence suggests a generalizable relationship between leakage and performance, though caution is warranted due to variability in accuracy outcomes across studies. These findings highlight the importance of methodological standards to reduce data leakage and promote transparency in neonatal EEG seizure detection models.