The Language-Modeling Cost Along the Base-to-Instruct Weight Interpolation Path of Qwen2.5-0.5B: WikiText-2 NLL versus ARC, HellaSwag and WinoGrande

Linear interpolation between a base language model and its instruction-tuned version, θ(t)=(1-t),θ_base+t,θ_inst, is often tuned for task accuracy alone. We measure what the same path does to language-modeling quality. For Qwen2.5-0.5B and its Instruct version, we evaluate 11 coefficients (t=0,0.1,…,1) on a macro-average of four multiple-choice benchmarks (3967 test items) and on WikiText-2 negative log-likelihood (NLL; 100 test blocks). All selection uses a disjoint validation split. The pre-registered hypothesis was that the accuracy-optimal t^* differs from the NLL-optimal one. It holds only literally: validation picks t^*_NLL=0.1 and t^*_acc=0.7, but t^*_NLL=0.1 is not distinguishable from the base (the interval contains zero) and t^*_acc=0.7 rests on about one net validation item (0.0008). Test NLL rises from t≈0.2 onward, reaching +0.0886 nats per token [+0.0831, +0.0939] at t=1. Accuracy shows no detectable difference from the base (+0.65 pp, 95% CI [-0.23,+1.52]). Evaluation in bfloat16 agrees with fp32 to within 0.003 nats. The study covers one model pair, one scale and one path.Preprint. The attached archive contains the LaTeX source, the per-item evaluation caches, all analysis scripts and the model revisions needed to regenerate every number, table and figure (git tag preprints-v1; entry point run_everything.sh). Author: Dheiver Francisco Santos, ORCID 0000-0002-8599-9436. License: CC BY 4.0.

Authors

Publication Details

Journal
Figshare
Published
2026-11-15
DOI
https://doi.org/10.6084/m9.figshare.35014015
Primary Topic
Topic Modeling
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

The Language-Modeling Cost Along the Base-to-Instruct Weight Interpolation Path of Qwen2.5-0.5B: WikiText-2 NLL versus ARC, HellaSwag and WinoGrande

Dheiver Francisco Santos
Figshare
Topic Modeling
preprint

The Language-Modeling Cost Along the Base-to-Instruct Weight Interpolation Path of Qwen2.5-0.5B: WikiText-2 NLL versus ARC, HellaSwag and WinoGrande

Dheiver Francisco Santos
preprint en

Abstract

Linear interpolation between a base language model and its instruction-tuned version, θ(t)=(1-t),θ_base+t,θ_inst, is often tuned for task accuracy alone. We measure what the same path does to language-modeling quality. For Qwen2.5-0.5B and its Instruct version, we evaluate 11 coefficients (t=0,0.1,…,1) on a macro-average of four multiple-choice benchmarks (3967 test items) and on WikiText-2 negative log-likelihood (NLL; 100 test blocks). All selection uses a disjoint validation split. The pre-registered hypothesis was that the accuracy-optimal t^* differs from the NLL-optimal one. It holds only literally: validation picks t^*_NLL=0.1 and t^*_acc=0.7, but t^*_NLL=0.1 is not distinguishable from the base (the interval contains zero) and t^*_acc=0.7 rests on about one net validation item (0.0008). Test NLL rises from t≈0.2 onward, reaching +0.0886 nats per token [+0.0831, +0.0939] at t=1. Accuracy shows no detectable difference from the base (+0.65 pp, 95% CI [-0.23,+1.52]). Evaluation in bfloat16 agrees with fp32 to within 0.003 nats. The study covers one model pair, one scale and one path.Preprint. The attached archive contains the LaTeX source, the per-item evaluation caches, all analysis scripts and the model revisions needed to regenerate every number, table and figure (git tag preprints-v1; entry point run_everything.sh). Author: Dheiver Francisco Santos, ORCID 0000-0002-8599-9436. License: CC BY 4.0.

Figshare
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon