How Uncertain Is a Cross-Validation Score? The Three-Term Variance of K-Fold CV, Why It Cannot Be Estimated Without Bias, and What Intervals Actually Cover

K-fold cross-validation is the default way to report a classifier's error, usually with a standard error computed as if the fold errors were independent. They are not. I prove that the variance of the K-fold estimate is the sum of three terms, the per-observation variance, the within-fold covariance and the between-fold covariance, and give a self-contained proof, checked symbolically, that no estimator built from the usual sums of squares is unbiased for it, including in the special case of 0-1 loss. I then measure the three components directly by drawing 800 independent training sets of 300 observations from two public UCI datasets, Spambase and MAGIC gamma telescope, and running 10-fold cross-validation with logistic regression and a decision tree. The between-fold covariance, which the naive formula ignores, accounts for 9% to 38% of the variance. The naive fold-variance estimate captures only 58% to 90% of the true variance, and its nominal 95% interval covers the mean cross-validation error in 87% to 93% of replicates. The Nadeau-Bengio correction overestimates the variance by a factor of 1.22 to 1.90 and covers 96% to 98%. Across replicates, the correlation between the cross-validation error and the true error of the model actually fitted is only 0.14 to 0.39: cross-validation measures how well the procedure does on average, not how well this model does.