How Many Validation Items Does Merge-Coefficient Selection Need? Selection Regret on Qwen2.5-0.5B Interpolations (ARC, HellaSwag, WinoGrande)

How large must the validation set be when selecting a merge coefficient? We take an 11-point linear-interpolation grid between Qwen2.5-0.5B and its Instruct variant, scored by the macro average of four multiple-choice benchmarks. We measure selection regret (test accuracy of the best grid point minus that of the selected point) on random validation subsets of n∈30,60,100,200 items per task, 1000 draws each, from a pool of about 300 items per task. The mean regret is 0.50, 0.53, 0.51 and 0.50 pp. We detect no trend, and the pre-registered hypothesis that regret falls with n is not supported. Selection does not change the expected regret, although the probability of zero regret roughly doubles (0.10 to 0.20). The probability of beating both originals (0.31–0.37) is of the same order as that of a random pick (0.36), whose regret is 0.46 pp. The test accuracies of all 11 points lie within 1.06 pp; we conjecture that this flatness sets the regret. One model pair, one grid, one pool.Preprint. The attached archive contains the LaTeX source, the per-item evaluation caches, all analysis scripts and the model revisions needed to regenerate every number, table and figure (git tag preprints-v1; entry point run_everything.sh). Author: Dheiver Francisco Santos, ORCID 0000-0002-8599-9436. License: CC BY 4.0.

Authors

Publication Details

Journal
Figshare
Published
2026-11-15
DOI
https://doi.org/10.6084/m9.figshare.35014009
Primary Topic
Topic Modeling
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

How Many Validation Items Does Merge-Coefficient Selection Need? Selection Regret on Qwen2.5-0.5B Interpolations (ARC, HellaSwag, WinoGrande)

Dheiver Francisco Santos
Figshare
Topic Modeling
preprint

How Many Validation Items Does Merge-Coefficient Selection Need? Selection Regret on Qwen2.5-0.5B Interpolations (ARC, HellaSwag, WinoGrande)

Dheiver Francisco Santos
preprint en

Abstract

How large must the validation set be when selecting a merge coefficient? We take an 11-point linear-interpolation grid between Qwen2.5-0.5B and its Instruct variant, scored by the macro average of four multiple-choice benchmarks. We measure selection regret (test accuracy of the best grid point minus that of the selected point) on random validation subsets of n∈30,60,100,200 items per task, 1000 draws each, from a pool of about 300 items per task. The mean regret is 0.50, 0.53, 0.51 and 0.50 pp. We detect no trend, and the pre-registered hypothesis that regret falls with n is not supported. Selection does not change the expected regret, although the probability of zero regret roughly doubles (0.10 to 0.20). The probability of beating both originals (0.31–0.37) is of the same order as that of a random pick (0.36), whose regret is 0.46 pp. The test accuracies of all 11 points lie within 1.06 pp; we conjecture that this flatness sets the regret. One model pair, one grid, one pool.Preprint. The attached archive contains the LaTeX source, the per-item evaluation caches, all analysis scripts and the model revisions needed to regenerate every number, table and figure (git tag preprints-v1; entry point run_everything.sh). Author: Dheiver Francisco Santos, ORCID 0000-0002-8599-9436. License: CC BY 4.0.

Figshare
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon