Not a panacea: the role of presenting descriptive statistics and effect sizes in abstracts on medical students’ misconceptions about p-values

Background Most scientific studies report inferential statistical results using p -values. For clinical decision-making, correctly interpreting the results of significance tests is particularly important, for example, to differentiate between statistical significance and clinical relevance. However, p -values are frequently misinterpreted, and reporting effect sizes alongside p -values has therefore been recommended to support the interpretation of statistical results. Methods We examined 121 medical students’ misconceptions about p -values when presented with three fictitious but realistic medical abstracts containing comparisons of group means and t -tests. Medical context, sample size, p -value, and effect size were systematically varied across the abstracts (abstract 1: N 1 = N 2 = 7,500; p < 0.001; Cohen’s d = 0.12; abstract 2: N 1 = N 2 = 10; p = 0.04; Cohen’s d = 1.00; abstract 3: N 1 = N 2 = 40; p = 0.09; Cohen’s d = 0.38). We varied the amount of statistical information provided in each abstract (information set 1: sample size and p -value; information set 2: sample size, p -value, means, and standard deviations; information set 3: the same information and effect size Cohen’s d ). We examined whether additional statistical information was associated with fewer misconceptions and how the provided information influenced students’ perceived effect size. Results In all versions, a high proportion of medical students across all semesters of study demonstrated multiple misconceptions about p -values. The number of endorsed misconceptions substantially decreased with the provision of more statistical information only in the context with a large sample size, a statistically significant result, and a very small effect size. In this context, providing additional statistical information reduced the tendency to infer clinical relevance and effect magnitude directly from statistical significance. Conclusions Although reporting standardized effect sizes has repeatedly been recommended, their inclusion alone appears insufficient to prevent mis- and overinterpretations of inferential statistical results. The interpretation of statistical evidence should be addressed more intensively and longitudinally throughout medical education, including by explicitly addressing typical misconceptions. Alternative teaching approaches to statistical inference, including simulation-based approaches, should also be considered.

Authors

Institutions

Publication Details

Journal
BMC Medical Research Methodology
Published
2026-11-19
DOI
https://doi.org/10.1186/s12874-026-03075-w
Primary Topic
Statistics Education and Methodologies
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Not a panacea: the role of presenting descriptive statistics and effect sizes in abstracts on medical students’ misconceptions about p-values

Ralf Schmidmaier, Karin Binder, Martin Rudolf Fischer, Leon Wiemann et al.
BMC Medical Research Methodology
Statistics Education and Methodologies
article

Not a panacea: the role of presenting descriptive statistics and effect sizes in abstracts on medical students’ misconceptions about p-values

Ralf Schmidmaier, Karin Binder, Martin Rudolf Fischer, Leon Wiemann, Leah Braun, Frieder Sebastian Eisenhut
article en

Abstract

Background Most scientific studies report inferential statistical results using p -values. For clinical decision-making, correctly interpreting the results of significance tests is particularly important, for example, to differentiate between statistical significance and clinical relevance. However, p -values are frequently misinterpreted, and reporting effect sizes alongside p -values has therefore been recommended to support the interpretation of statistical results. Methods We examined 121 medical students’ misconceptions about p -values when presented with three fictitious but realistic medical abstracts containing comparisons of group means and t -tests. Medical context, sample size, p -value, and effect size were systematically varied across the abstracts (abstract 1: N 1 = N 2 = 7,500; p < 0.001; Cohen’s d = 0.12; abstract 2: N 1 = N 2 = 10; p = 0.04; Cohen’s d = 1.00; abstract 3: N 1 = N 2 = 40; p = 0.09; Cohen’s d = 0.38). We varied the amount of statistical information provided in each abstract (information set 1: sample size and p -value; information set 2: sample size, p -value, means, and standard deviations; information set 3: the same information and effect size Cohen’s d ). We examined whether additional statistical information was associated with fewer misconceptions and how the provided information influenced students’ perceived effect size. Results In all versions, a high proportion of medical students across all semesters of study demonstrated multiple misconceptions about p -values. The number of endorsed misconceptions substantially decreased with the provision of more statistical information only in the context with a large sample size, a statistically significant result, and a very small effect size. In this context, providing additional statistical information reduced the tendency to infer clinical relevance and effect magnitude directly from statistical significance. Conclusions Although reporting standardized effect sizes has repeatedly been recommended, their inclusion alone appears insufficient to prevent mis- and overinterpretations of inferential statistical results. The interpretation of statistical evidence should be addressed more intensively and longitudinally throughout medical education, including by explicitly addressing typical misconceptions. Alternative teaching approaches to statistical inference, including simulation-based approaches, should also be considered.

BMC Medical Research MethodologyVol. 26(1)
Paderborn University (DE), LMU Klinikum (DE), Institut für Didaktik und Ausbildungsforschung in der Medizin (DE), Ludwig-Maximilians-Universität München (DE)
Good health and well-being
Statistics Education and Methodologies
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon