Winner's Curse in Validation-Selected Model Merges: A Resampling Study on the Qwen2.5-0.5B Interpolation Grid (ARC, HellaSwag, WinoGrande)

Merge coefficients are usually picked on a small validation set. The maximum over many candidates inflates the winner's validation score (the winner's curse); we did not find a measurement of this inflation for model merging. We measure it on 11 linear interpolations θ_t=(1-t)θ_base+t,θ_inst between Qwen2.5-0.5B and Qwen2.5-0.5B-Instruct, scored in bfloat16 by the macro average of four benchmarks (ARC-Easy, ARC-Challenge, HellaSwag, WinoGrande; 300 validation and about 1000 test items per task). We select t on 1000 bootstrap resamples of the validation set and compare the validation gain of the selected t over t=0 with its test gain. The mean validation gain is +0.64 pp and the mean test gain is +0.17 pp, a ratio of 3.8. The mean difference, +0.48 pp, comes from simple resampling. The double bootstrap (test items resampled too) gives a 95% dispersion interval of [-0.65,+1.77] pp, and 67% of draws are positive. The pure selection bias is +0.68 pp. The data are consistent with a winner's curse; its magnitude is unresolved and conditional on bfloat16. One model pair, one grid.Preprint. The attached archive contains the LaTeX source, the per-item evaluation caches, all analysis scripts and the model revisions needed to regenerate every number, table and figure (git tag preprints-v1; entry point run_everything.sh). Author: Dheiver Francisco Santos, ORCID 0000-0002-8599-9436. License: CC BY 4.0.

Authors

Publication Details

Journal
Figshare
Published
2026-11-15
DOI
https://doi.org/10.6084/m9.figshare.35014006
Primary Topic
Domain Adaptation and Few-Shot Learning
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Winner's Curse in Validation-Selected Model Merges: A Resampling Study on the Qwen2.5-0.5B Interpolation Grid (ARC, HellaSwag, WinoGrande)

Dheiver Francisco Santos
Figshare
Domain Adaptation and Few-Shot Learning
preprint

Winner's Curse in Validation-Selected Model Merges: A Resampling Study on the Qwen2.5-0.5B Interpolation Grid (ARC, HellaSwag, WinoGrande)

Dheiver Francisco Santos
preprint en

Abstract

Merge coefficients are usually picked on a small validation set. The maximum over many candidates inflates the winner's validation score (the winner's curse); we did not find a measurement of this inflation for model merging. We measure it on 11 linear interpolations θ_t=(1-t)θ_base+t,θ_inst between Qwen2.5-0.5B and Qwen2.5-0.5B-Instruct, scored in bfloat16 by the macro average of four benchmarks (ARC-Easy, ARC-Challenge, HellaSwag, WinoGrande; 300 validation and about 1000 test items per task). We select t on 1000 bootstrap resamples of the validation set and compare the validation gain of the selected t over t=0 with its test gain. The mean validation gain is +0.64 pp and the mean test gain is +0.17 pp, a ratio of 3.8. The mean difference, +0.48 pp, comes from simple resampling. The double bootstrap (test items resampled too) gives a 95% dispersion interval of [-0.65,+1.77] pp, and 67% of draws are positive. The pure selection bias is +0.68 pp. The data are consistent with a winner's curse; its magnitude is unresolved and conditional on bfloat16. One model pair, one grid.Preprint. The attached archive contains the LaTeX source, the per-item evaluation caches, all analysis scripts and the model revisions needed to regenerate every number, table and figure (git tag preprints-v1; entry point run_everything.sh). Author: Dheiver Francisco Santos, ORCID 0000-0002-8599-9436. License: CC BY 4.0.

Figshare
Domain Adaptation and Few-Shot Learning
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon