Reasoning Again with a Quarter of the Experts: Retry First, Then Train

Removing three quarters of the routed experts from a 122B-parameter reasoning mixture-of-experts model makes it fit in a 12 GiB file, and breaks it in a specific way: on LiveCodeBench it enters verbatim reasoning loops that run to the token limit on 44 of 120 development problems, where the unpruned model at the same 2-bit quantization loops on none. We study how much of that loss can be recovered without changing the kept experts' quantized bytes, in three steps of increasing cost. On development problems, a sampling penalty recovers part of it. Stopping a reply when it starts to loop and drawing again recovers more on top of it, with no weight change. Our method, B, adds rank-8 trained corrections to the kept experts, loaded as an adapter, and restores two experts per layer; it is fitted layer by layer to the unpruned model's expert block on the pruned model's own activations. In a pre-registered comparison on 130 easy and medium LiveCodeBench problems that no part of the study had touched, with both arms given the same sampling penalty and the same stop-and-redraw policy, B solves 116 problems against 104 (+9.2 pp, 95% CI [+3.1, +15.4]; 15 problems gained, 3 lost). A training-free weight edit aimed at the loops, tested the same way earlier, did not help. B removes more wrong answers than loops, and costs 1.7x the wall time. On two older benchmarks used as regression checks, B stays inside a pre-set 2 pp tolerance but shows no gain (HumanEval+ +0.6 pp, MBPP+ -1.9 pp). The evidence is one model, one quantization and one benchmark family; we state what it does and does not show.

Authors

Publication Details

Journal
arXiv (Cornell University)
Published
2026-11-16
DOI
https://doi.org/10.48550/arxiv.2611.20419
Primary Topic
Topic Modeling
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Reasoning Again with a Quarter of the Experts: Retry First, Then Train

Anik Jha
arXiv (Cornell University)
Topic Modeling
preprint

Reasoning Again with a Quarter of the Experts: Retry First, Then Train

Anik Jha
preprint en

Abstract

Removing three quarters of the routed experts from a 122B-parameter reasoning mixture-of-experts model makes it fit in a 12 GiB file, and breaks it in a specific way: on LiveCodeBench it enters verbatim reasoning loops that run to the token limit on 44 of 120 development problems, where the unpruned model at the same 2-bit quantization loops on none. We study how much of that loss can be recovered without changing the kept experts' quantized bytes, in three steps of increasing cost. On development problems, a sampling penalty recovers part of it. Stopping a reply when it starts to loop and drawing again recovers more on top of it, with no weight change. Our method, B, adds rank-8 trained corrections to the kept experts, loaded as an adapter, and restores two experts per layer; it is fitted layer by layer to the unpruned model's expert block on the pruned model's own activations. In a pre-registered comparison on 130 easy and medium LiveCodeBench problems that no part of the study had touched, with both arms given the same sampling penalty and the same stop-and-redraw policy, B solves 116 problems against 104 (+9.2 pp, 95% CI [+3.1, +15.4]; 15 problems gained, 3 lost). A training-free weight edit aimed at the loops, tested the same way earlier, did not help. B removes more wrong answers than loops, and costs 1.7x the wall time. On two older benchmarks used as regression checks, B stays inside a pre-set 2 pp tolerance but shows no gain (HumanEval+ +0.6 pp, MBPP+ -1.9 pp). The evidence is one model, one quantization and one benchmark family; we state what it does and does not show.

arXiv (Cornell University)
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon