Counteracting the overestimation of effect size in mass-univariate brain-wide association studies using cross-validation

Abstract Background Brain-wide association effects are commonly overestimated in psychiatric neuroimaging research, leading to low replicability and generalizability of findings, and ultimately to poor translation to clinical utility. Cross-validation is currently not standardly used in univariate neuroimaging analyses but could pose a potential counteracting measure for this problem. Methods We systematically evaluated the benefit of a cross-validation procedure to expose inflated effect size estimates in mass-univariate voxel-based analysis, using body mass index (BMI) as an example predictor for gray matter. A total of n = 3401 MRI scans from a psychiatric cohort were used to investigate the impact of applying cross-validation on effect size estimates across a range of sample sizes and across different localizations with a range of underlying pooled effect sizes. Results BMI was negatively associated with gray matter density with a maximum underlying effect size of R ² p = 0.036 (in the full sample of n = 3401). However, without cross-validation, effects were substantially overestimated in small samples, with effect sizes ranging up to R ² p = 0.535. Even with an underlying null-effect, effect size estimates up to R ² p = 0.429 were reached in samples with n = 25. Applying cross-validation revealed that these inflated effects in small samples did not generalize to novel data. A minimum sample size of n = 100 was required for largest brain-wide effects to start generalizing, while samples with n = 400 were needed for smaller effects to generalize. This ‘point of initial generalizability’ was not reached for underlying null effects, even with the maximum available sample of n = 3401. Conclusions Results suggest that cross-validation is an effective measure to expose overestimated effects in small samples and thus could help to counteract low replicability and generalizability in brain-wide mass-univariate association studies. Open-source code is provided for practical implementation.

Authors

Institutions

Publication Details

Journal
Translational Psychiatry
Published
2026-11-04
DOI
https://doi.org/10.1038/s41398-026-04529-x
Citations
7
Primary Topic
Functional Brain Connectivity Studies
Type
article
Field-Weighted Citation Impact
12.10

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Counteracting the overestimation of effect size in mass-univariate brain-wide association studies using cross-validation

Adrian Wroblewski, L. Fisch, Andreas Jansen, Kira Flinkenflügel et al.
7 citations
Translational Psychiatry
Functional Brain Connectivity Studies
12.10
article

Counteracting the overestimation of effect size in mass-univariate brain-wide association studies using cross-validation

Adrian Wroblewski, L. Fisch, Andreas Jansen, Kira Flinkenflügel, Janik Goltermann, Elisabeth Johanna Leehr, Marius Gruber, Dominik Grotegerd, Susanne L. Meinert, Alexandra Winter, Katharina Dohm, Katharina Thiel, Ramona Leenings, Hamidreza Jamalabadi, Nils Ralf Winter, Maike Frederike Richter, Jan Ernsting, Benjamin Straube, Florian Thomas‐Odenthal, Nina Alexander, Tim Hahn, Igor Nenadić, Nils Opel, Frederike Stein, Paula L Usemann, Klaus Berger, Udo Dannlowski, Tilo T. J. Kircher, Joscha Böhnlein, Heike Minnerup, Anna Kraus, Carlotta Barkhau, Katharina Brosch, Julia‐Katharina Pfarr, Christian Gaser B, Lea Teutenberg
article en
7 citations

Abstract

Abstract Background Brain-wide association effects are commonly overestimated in psychiatric neuroimaging research, leading to low replicability and generalizability of findings, and ultimately to poor translation to clinical utility. Cross-validation is currently not standardly used in univariate neuroimaging analyses but could pose a potential counteracting measure for this problem. Methods We systematically evaluated the benefit of a cross-validation procedure to expose inflated effect size estimates in mass-univariate voxel-based analysis, using body mass index (BMI) as an example predictor for gray matter. A total of n = 3401 MRI scans from a psychiatric cohort were used to investigate the impact of applying cross-validation on effect size estimates across a range of sample sizes and across different localizations with a range of underlying pooled effect sizes. Results BMI was negatively associated with gray matter density with a maximum underlying effect size of R ² p = 0.036 (in the full sample of n = 3401). However, without cross-validation, effects were substantially overestimated in small samples, with effect sizes ranging up to R ² p = 0.535. Even with an underlying null-effect, effect size estimates up to R ² p = 0.429 were reached in samples with n = 25. Applying cross-validation revealed that these inflated effects in small samples did not generalize to novel data. A minimum sample size of n = 100 was required for largest brain-wide effects to start generalizing, while samples with n = 400 were needed for smaller effects to generalize. This ‘point of initial generalizability’ was not reached for underlying null effects, even with the maximum available sample of n = 3401. Conclusions Results suggest that cross-validation is an effective measure to expose overestimated effects in small samples and thus could help to counteract low replicability and generalizability in brain-wide mass-univariate association studies. Open-source code is provided for practical implementation.

Translational PsychiatryVol. 16(1)
Philipps University of Marburg (DE), University of Münster (DE), Jena University Hospital (DE), Deutsches Zentrum für Psychische Gesundheit (DE)
Deutsche Forschungsgemeinschaft, Nederlandse Organisatie voor Wetenschappelijk Onderzoek, Servier
Good health and well-being
Openalex Percentile: Top 2%
Functional Brain Connectivity Studies
12.10
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon