Agentic Weight-Space Programming as an Alternative to Fine-Tuning in Small Language Models

Small reasoning language models often fail by looping: with thinking on and a 12,288-token budget, Qwen3.5-0.8B runs out of space before answering on about 85% of grade-school math problems. We ask whether frontier-model agents can repair this defect by editing the weights directly, without gradients, and how the result compares with simpler fixes. Claude agents, given the weights, a small testing budget and at most four rank-1 edits, found fixes in about two hours per run on a single consumer GPU (GTX 1660 SUPER, 6 GB). The edits raise accuracy on locked GSM8K test problems from 15.5% to 61.5–68% (two locked sets of 200, the second with every arm and prediction registered before opening), and six runs of the same protocol all reached 58–67% on practice problems. Independent agents found the same “Wait”-brake neurons and the same lever. The edit that GPT-6 Astra wrote in Sparks of Weight-Space Programming (Takehi et al., MIT CSAIL, 2026) reaches the same accuracy only with a 20,480-token budget, so the agents’ advantage over it is efficiency: 1.7–1.9× fewer tokens. The agents’ edits also beat a LoRA fine-tuning baseline that used 13× more GPU time. Simple alternatives are strong: over the 400 locked problems, switching thinking off is only 6.5–9 points behind the edits at a third of their tokens, and ties the reduced edit. A scripted search with the same tools but no agent reasoning failed, every cheap proxy score was eventually gamed, and more interpretability context hurt. On a diffuse defect (instruction following, IFEval), neither the agents, nor prompting, nor fine-tuning helped. Every prediction was written in a dated lab notebook before the result. The PDF is provided in English and in Brazilian Portuguese.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-13
DOI
https://doi.org/10.5281/zenodo.24208534
Primary Topic
Topic Modeling
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Agentic Weight-Space Programming as an Alternative to Fine-Tuning in Small Language Models

Luiz Felipe De Melo
4 links
Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
preprint

Agentic Weight-Space Programming as an Alternative to Fine-Tuning in Small Language Models

Luiz Felipe De Melo
preprint en

Abstract

Small reasoning language models often fail by looping: with thinking on and a 12,288-token budget, Qwen3.5-0.8B runs out of space before answering on about 85% of grade-school math problems. We ask whether frontier-model agents can repair this defect by editing the weights directly, without gradients, and how the result compares with simpler fixes. Claude agents, given the weights, a small testing budget and at most four rank-1 edits, found fixes in about two hours per run on a single consumer GPU (GTX 1660 SUPER, 6 GB). The edits raise accuracy on locked GSM8K test problems from 15.5% to 61.5–68% (two locked sets of 200, the second with every arm and prediction registered before opening), and six runs of the same protocol all reached 58–67% on practice problems. Independent agents found the same “Wait”-brake neurons and the same lever. The edit that GPT-6 Astra wrote in Sparks of Weight-Space Programming (Takehi et al., MIT CSAIL, 2026) reaches the same accuracy only with a 20,480-token budget, so the agents’ advantage over it is efficiency: 1.7–1.9× fewer tokens. The agents’ edits also beat a LoRA fine-tuning baseline that used 13× more GPU time. Simple alternatives are strong: over the 400 locked problems, switching thinking off is only 6.5–9 points behind the edits at a third of their tokens, and ties the reduced edit. A scripted search with the same tools but no agent reasoning failed, every cheap proxy score was eventually gamed, and more interpretability context hurt. On a diffuse defect (instruction following, IFEval), neither the agents, nor prompting, nor fine-tuning helped. Every prediction was written in a dated lab notebook before the result. The PDF is provided in English and in Brazilian Portuguese.

Zenodo (CERN European Organization for Nuclear Research)
Universidade do Sul de Santa Catarina (BR)
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon