Which Way do Small-Sample Confidence Intervals for a Difference of Means Fail? Coverage, Directional Error and the Signed Skewness Balance

Background: The difference between two arithmetic means remains a central target in small independent-group studies, yet total interval coverage does not say on which side an interval fails. We evaluated coverage, the side of noncoverage and interval width for 95% intervals, and tested a predeclared hypothesis about the direction of failure. Methods: A predeclared simulation study (plan dated before the first run) compared Welch, percentile-bootstrap, basic-bootstrap, bootstrap-t and BCa intervals in 147 settings with 10,000 simulated datasets and 1,999 separate-group resamples each. Seven population families (including left-skewed and opposed-skew designs, so that direction is not built into the design), seven sample-size allocations from (5,5) to (100,100) and variance ratios 1, 4 and 10 were crossed. The primary outcomes were coverage, the probability L that the interval lies entirely below the true difference, the probability U that it lies entirely above it, and their difference. The predeclared hypothesis was that the sign of L-U follows the sign of Γ, the standardised third cumulant of the difference in sample means. Earlier exploratory stages are reported separately and not pooled. Results: For Welch intervals the sign of L-U agreed with the sign of Γ in 66 of 68 determinate settings (97.1%); none of 78 settings with Γ=0 showed imbalance beyond three Monte Carlo standard errors. For right-skewed lognormal data with variance ratio 1 the failure was on the lower side when the small group was Group 1 (+14.4 percentage points) and on the upper side when it was Group 2 (-14.2). With opposed skew in the two groups, imbalance was +14.4 points at n=5 and +5.8 at n=100 per group, against about zero for same-shape skew. The first-order size of the imbalance was over-predicted (slope 0.56 for groups of at least 30). Worst-case Welch coverage was 68.1% (varying-shape lognormal) and 80.9% (fixed-shape lognormal). Bootstrap-t halved the median imbalance at group sizes of 30 and above (median ratio to Welch 0.53) but not at group sizes of 10 or below. Percentile, basic and BCa intervals under-covered even for normal data at n≤10 (85.6–92.2% across such settings). Conclusions: In small samples the side on which an interval fails is systematic and predictable in sign from the balance of skewness between groups weighted by their contribution to the variance of the difference, whereas its size is not captured by the first-order term. No procedure was uniformly reliable. Studies should report lower- and upper-side noncoverage, not total coverage alone.

Authors

Publication Details

Journal
Preprints.org
Published
2026-11-18
DOI
https://doi.org/10.20944/preprints202611.1797.v1
Primary Topic
Probability and Statistical Research
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Which Way do Small-Sample Confidence Intervals for a Difference of Means Fail? Coverage, Directional Error and the Signed Skewness Balance

Johan G.L. Verheyden
Preprints.org
Probability and Statistical Research
preprint

Which Way do Small-Sample Confidence Intervals for a Difference of Means Fail? Coverage, Directional Error and the Signed Skewness Balance

Johan G.L. Verheyden
preprint en

Abstract

Background: The difference between two arithmetic means remains a central target in small independent-group studies, yet total interval coverage does not say on which side an interval fails. We evaluated coverage, the side of noncoverage and interval width for 95% intervals, and tested a predeclared hypothesis about the direction of failure. Methods: A predeclared simulation study (plan dated before the first run) compared Welch, percentile-bootstrap, basic-bootstrap, bootstrap-t and BCa intervals in 147 settings with 10,000 simulated datasets and 1,999 separate-group resamples each. Seven population families (including left-skewed and opposed-skew designs, so that direction is not built into the design), seven sample-size allocations from (5,5) to (100,100) and variance ratios 1, 4 and 10 were crossed. The primary outcomes were coverage, the probability L that the interval lies entirely below the true difference, the probability U that it lies entirely above it, and their difference. The predeclared hypothesis was that the sign of L-U follows the sign of Γ, the standardised third cumulant of the difference in sample means. Earlier exploratory stages are reported separately and not pooled. Results: For Welch intervals the sign of L-U agreed with the sign of Γ in 66 of 68 determinate settings (97.1%); none of 78 settings with Γ=0 showed imbalance beyond three Monte Carlo standard errors. For right-skewed lognormal data with variance ratio 1 the failure was on the lower side when the small group was Group 1 (+14.4 percentage points) and on the upper side when it was Group 2 (-14.2). With opposed skew in the two groups, imbalance was +14.4 points at n=5 and +5.8 at n=100 per group, against about zero for same-shape skew. The first-order size of the imbalance was over-predicted (slope 0.56 for groups of at least 30). Worst-case Welch coverage was 68.1% (varying-shape lognormal) and 80.9% (fixed-shape lognormal). Bootstrap-t halved the median imbalance at group sizes of 30 and above (median ratio to Welch 0.53) but not at group sizes of 10 or below. Percentile, basic and BCa intervals under-covered even for normal data at n≤10 (85.6–92.2% across such settings). Conclusions: In small samples the side on which an interval fails is systematic and predictable in sign from the balance of skewness between groups weighted by their contribution to the variance of the difference, whereas its size is not captured by the first-order term. No procedure was uniformly reliable. Studies should report lower- and upper-side noncoverage, not total coverage alone.

Preprints.org
Probability and Statistical Research
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon