Mind the Gap: Explicit Cross-Benchmark Calibrations for Estimating Missing LLM Evaluation Scores

Abstract Most language models are evaluated on only a few benchmarks: in our compilation of 401 models and 146 benchmark versions, the benchmark test coverage is only 8.3% on the most frequently used benchmarks. We present benchgap, a pipeline that estimates the missing scores without running any evaluations. It calibrates each benchmark against each same-capability benchmark on the models measured on both, fitting six monotone curve families and keeping the one with the lowest leave-one-out cross-validated error; multivariate mappings combine several sources. Calibrations never cross capability groups, must rise and never extrapolate beyond one fitted-range width; estimates never feed further estimates, and each carries its own cross-validated error and confidence level. In an end-to-end holdout of 15% of the measured scores, the pipeline fills 71% of the masked cells at 4.6 percentage points (pp) mean absolute error (MAE), 23% lower than a rank-2 matrix-completion baseline on the same cells; on the cells it leaves empty, the baseline's MAE is 11.8 pp. Realized MAE rises with the confidence level (3.1, 5.1 and 7.5 pp from high to low), but the realized root-mean-square error exceeds the claimed error labels, which are selected minima, by a factor of 1.4–1.7. We also quantify the harness tax, the disagreement between evaluation setups: on verified same-item-set pairs it is 8.7 pp on agentic benchmarks against 2.3 pp on tool-free ones, and enabling tools on Humanity's Last Exam raises the scores of all 11 models measured both ways, by 10.0 pp on average. Estimates are updated daily at https://benchgap.net.