The Language-Modeling Cost Along the Base-to-Instruct Weight Interpolation Path of Qwen2.5-0.5B: WikiText-2 NLL versus ARC, HellaSwag and WinoGrande
Linear interpolation between a base language model and its instruction-tuned version, θ(t)=(1-t),θ_base+t,θ_inst, is often tuned for task accuracy alone. We measure what the same path does to language-modeling quality. For Qwen2.5-0.5B and its Instruct version, we evaluate 11 coefficients (t=0,0.1,…,1) on a macro-average of four multiple-choice benchmarks (3967 test items) and on WikiText-2 negative log-likelihood (NLL; 100 test blocks). All selection uses a disjoint validation split. The pre-registered hypothesis was that the accuracy-optimal t^* differs from the NLL-optimal one. It holds only literally: validation picks t^*_NLL=0.1 and t^*_acc=0.7, but t^*_NLL=0.1 is not distinguishable from the base (the interval contains zero) and t^*_acc=0.7 rests on about one net validation item (0.0008). Test NLL rises from t≈0.2 onward, reaching +0.0886 nats per token [+0.0831, +0.0939] at t=1. Accuracy shows no detectable difference from the base (+0.65 pp, 95% CI [-0.23,+1.52]). Evaluation in bfloat16 agrees with fp32 to within 0.003 nats. The study covers one model pair, one scale and one path.Preprint. The attached archive contains the LaTeX source, the per-item evaluation caches, all analysis scripts and the model revisions needed to regenerate every number, table and figure (git tag preprints-v1; entry point run_everything.sh). Author: Dheiver Francisco Santos, ORCID 0000-0002-8599-9436. License: CC BY 4.0.
Authors
- Dheiver Francisco Santos (ORCID: https://orcid.org/0000-0002-8599-9436)
Publication Details
- Journal
- Figshare
- Published
- 2026-11-15
- DOI
- https://doi.org/10.6084/m9.figshare.35014015
- Primary Topic
- Topic Modeling
- Type
- preprint