When 95% Isn't 95%: A Monte Carlo Investigation of Confidence Interval Reliability

A nominal 95% confidence interval is expected to contain the true population parameter in approximately 95% of repeated samples. However, actual coverage depends on the statistical method, sample size, and characteristics of the underlying population distribution.This study uses Monte Carlo simulation to compare the empirical coverage probabilities and average widths of conventional Student's t confidence intervals and percentile bootstrap confidence intervals for estimating a population mean. The investigation examines three population distributions—normal, moderately skewed gamma, and strongly skewed exponential—across four sample sizes (n = 10, 30, 100, and 500). Each of the 12 experimental conditions was evaluated using 1,000 Monte Carlo replications and 1,000 bootstrap resamples per replication.Across all 12 conditions, Student's t intervals achieved coverage closer to the nominal 95% level than percentile bootstrap intervals. Differences were particularly pronounced for small samples and skewed distributions. Although percentile bootstrap intervals were generally narrower, this reduction in interval width was accompanied by lower empirical coverage. Differences between the two methods diminished as sample size increased.These findings illustrate that bootstrap confidence intervals do not necessarily provide more reliable coverage under non-normality. The results emphasize the importance of considering sample size, population skewness, and interval construction methods when interpreting confidence intervals.

Authors

Publication Details

Journal
Figshare
Published
2026-11-15
DOI
https://doi.org/10.6084/m9.figshare.35016253
Primary Topic
Probability and Statistical Research
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

When 95% Isn't 95%: A Monte Carlo Investigation of Confidence Interval Reliability

Timothy Xeriland
Figshare
Probability and Statistical Research
preprint

When 95% Isn't 95%: A Monte Carlo Investigation of Confidence Interval Reliability

Timothy Xeriland
preprint en

Abstract

A nominal 95% confidence interval is expected to contain the true population parameter in approximately 95% of repeated samples. However, actual coverage depends on the statistical method, sample size, and characteristics of the underlying population distribution.This study uses Monte Carlo simulation to compare the empirical coverage probabilities and average widths of conventional Student's t confidence intervals and percentile bootstrap confidence intervals for estimating a population mean. The investigation examines three population distributions—normal, moderately skewed gamma, and strongly skewed exponential—across four sample sizes (n = 10, 30, 100, and 500). Each of the 12 experimental conditions was evaluated using 1,000 Monte Carlo replications and 1,000 bootstrap resamples per replication.Across all 12 conditions, Student's t intervals achieved coverage closer to the nominal 95% level than percentile bootstrap intervals. Differences were particularly pronounced for small samples and skewed distributions. Although percentile bootstrap intervals were generally narrower, this reduction in interval width was accompanied by lower empirical coverage. Differences between the two methods diminished as sample size increased.These findings illustrate that bootstrap confidence intervals do not necessarily provide more reliable coverage under non-normality. The results emphasize the importance of considering sample size, population skewness, and interval construction methods when interpreting confidence intervals.

Figshare
Probability and Statistical Research
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon