Randomization protects the average, not the trial: chance covariate imbalance and a calibration of balance P values at the design stage

Background. Randomization makes the unadjusted treatment-effect estimate unbiased on average over the allocations a scheme could produce, but does not balance any single trial. Design-stage tools increasingly score randomization schemes by simulated balance P values, yet what a balance P value implies for the bias and error rates of the eventual analysis is seldom stated. We review the evidence on chance imbalance and derive that link. Methods. A targeted narrative review of methodological, simulation, re-analysis and meta-epidemiological studies, regulatory guidance (FDA 2023, EMA 2015, ICH E9) and CONSORT 2025, with analytic derivations under a linear model with one normally distributed prognostic covariate. All results were computed in Python and checked by Monte Carlo simulation; derivations and code are supplied. Results. Given the realized allocation, the unadjusted estimate is biased by β × (difference in covariate means). In standard-error units it equals ρ × Zbal (ρ, the covariate–outcome correlation; Zbal, the balance z-statistic), so its distribution does not shrink as trials grow. With 10 independent covariates, 40% of trials have at least one baseline P < 0.05. Pre-specified ANCOVA removes this bias; change scores reverse its sign; non-collapsibility attenuates odds and hazard ratios even under perfect balance. Regulators converge on a few prognostic covariates, balanced by design with a random element and adjusted for. For design-stage scoring, a worst-balance threshold of P = 1/e is where the Sellke–Bayarri–Berger bound reaches 1. Under complete randomization the worst of k balance P values falls below 1/e with probability 1 − (1 − 1/e)k (0.99 for k = 10). Any balance P value ≥ 0.317 keeps the two-sided conditional type I error of the unadjusted test at or below 5% for every ρ; at 1/e the one-sided error in the favoured direction is at most about 4%, against 13–33% at P = 0.05 for ρ of 0.5–0.9. Conclusions. A balance score is a property of a scheme, interpretable only against complete randomization on the same covariates, weighted by prognostic strength, and reported alongside predictability and an analysis plan that adjusts for every balancing factor. The 1/e threshold is defensible but remains a design choice. Preprint, version 1.0. Not peer reviewed. Additional file 1 (derivations and supplementary tables and figures) is included; the code is archived separately (see Related works).

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-13
DOI
https://doi.org/10.5281/zenodo.24210824
Primary Topic
Statistical Methods in Clinical Trials
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Randomization protects the average, not the trial: chance covariate imbalance and a calibration of balance P values at the design stage

William Dwyer
Zenodo (CERN European Organization for Nuclear Research)
Statistical Methods in Clinical Trials
preprint

Randomization protects the average, not the trial: chance covariate imbalance and a calibration of balance P values at the design stage

William Dwyer
preprint en

Abstract

Background. Randomization makes the unadjusted treatment-effect estimate unbiased on average over the allocations a scheme could produce, but does not balance any single trial. Design-stage tools increasingly score randomization schemes by simulated balance P values, yet what a balance P value implies for the bias and error rates of the eventual analysis is seldom stated. We review the evidence on chance imbalance and derive that link. Methods. A targeted narrative review of methodological, simulation, re-analysis and meta-epidemiological studies, regulatory guidance (FDA 2023, EMA 2015, ICH E9) and CONSORT 2025, with analytic derivations under a linear model with one normally distributed prognostic covariate. All results were computed in Python and checked by Monte Carlo simulation; derivations and code are supplied. Results. Given the realized allocation, the unadjusted estimate is biased by β × (difference in covariate means). In standard-error units it equals ρ × Zbal (ρ, the covariate–outcome correlation; Zbal, the balance z-statistic), so its distribution does not shrink as trials grow. With 10 independent covariates, 40% of trials have at least one baseline P < 0.05. Pre-specified ANCOVA removes this bias; change scores reverse its sign; non-collapsibility attenuates odds and hazard ratios even under perfect balance. Regulators converge on a few prognostic covariates, balanced by design with a random element and adjusted for. For design-stage scoring, a worst-balance threshold of P = 1/e is where the Sellke–Bayarri–Berger bound reaches 1. Under complete randomization the worst of k balance P values falls below 1/e with probability 1 − (1 − 1/e)k (0.99 for k = 10). Any balance P value ≥ 0.317 keeps the two-sided conditional type I error of the unadjusted test at or below 5% for every ρ; at 1/e the one-sided error in the favoured direction is at most about 4%, against 13–33% at P = 0.05 for ρ of 0.5–0.9. Conclusions. A balance score is a property of a scheme, interpretable only against complete randomization on the same covariates, weighted by prognostic strength, and reported alongside predictability and an analysis plan that adjusts for every balancing factor. The 1/e threshold is defensible but remains a design choice. Preprint, version 1.0. Not peer reviewed. Additional file 1 (derivations and supplementary tables and figures) is included; the code is archived separately (see Related works).

Zenodo (CERN European Organization for Nuclear Research)
University of Massachusetts Lowell (US)
Statistical Methods in Clinical Trials
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon