Cross-cohort validation of EEG biomarkers using a pre-registered three-criterion framework: a demonstration in amyotrophic lateral sclerosis
Background. Despite decades of EEG biomarker development for amyotrophic lateral sclerosis (ALS), no candidate has progressed to clinical deployment, partly because cross-cohort validation is rare. We aimed to develop and demonstrate a pre-registered three-criterion screening framework to identify which candidate EEG features show the combined properties required for biomarker translation: cross-sectional discrimination, within-subject reliability, and cross-cohort effect consistency. Methods. We pre-registered a validation framework adapted from FDA-NIH BEST principles and applied it to 18 EEG features across three independent cohorts: EEGET-ALS (six ALS patients, 170 healthy controls; home recordings over 17 days), UCL/Penn State (eight ALS patients; clinic recordings over 1-2 months), and a healthy adult test-retest reference (60 participants, three sessions over one month). Features were required to meet three pre-registered criteria: (1) cross-sectional discrimination (Cohen's d > 0.5), (2) within-subject reliability (ICC(3,1) > 0.6 in both ALS cohorts independently), and (3) cross-cohort effect direction consistency. Cross-cohort permutation testing (1000 iterations) and subject-level bootstrap resampling (500 iterations) were applied to assess robustness. Results. Three features met all three pre-registered criteria: gamma power, theta power, and spectral entropy. Cross-cohort permutation testing showed that no permutation, out of 1000, produced any feature in the three-cohort reliability intersection (Monte Carlo p = 0.001). Bootstrap resampling indicated that the composition of the selected set was sensitive to sample composition (median count 1, 95% CI [0, 4]), with gamma power (67% inclusion), spectral entropy (43%), and theta power (32%) the most stable features at a relaxed reliability threshold (ICC > 0.5); at the primary pre-registered threshold (ICC > 0.6), the corresponding inclusion probabilities are lower (57%, 27%, and 19%, respectively). A principal methodological finding was that cross-cohort differences in feature reliability were large relative to the apparent disease-associated signal, most clearly demonstrated by relative beta power (ICC = 0.09 in a home-recorded cohort versus 0.93 in a clinic-recorded cohort); because recording protocol, hardware, setting, population, and preprocessing all differ jointly between cohorts in this dataset, this is evidence of cohort-associated variability that plausibly includes a protocol contribution, not a separately identified, isolated protocol effect. Conclusions. This study provides a pre-registered, multi-cohort EEG biomarker screening framework for ALS with explicit bootstrap-quantified uncertainty. Gamma power, spectral entropy, and theta power are identified as priority candidates for prospective validation. A recurring methodological observation is that protocol-related cohort differences appear to contribute more to cross-cohort variability than disease-related effects within this dataset; because protocol is confounded with cohort, equipment, and preprocessing, this pattern is descriptive rather than causal. The observation nonetheless has direct implications for the design of multi-site EEG biomarker studies in neural engineering.
Authors
- Haben Yhdego (ORCID: https://orcid.org/0000-0002-7243-0515)
- Tewolde Teklu (ORCID: https://orcid.org/0000-0002-8555-0591)
- Belay Gebregergis
Institutions
- University of Toronto (CA)
- Old Dominion University (US)
Publication Details
- Journal
- medRxiv
- Published
- 2026-11-12
- DOI
- https://doi.org/10.64898/2026.11.10.26367741
- Primary Topic
- Amyotrophic Lateral Sclerosis Research
- Type
- preprint