Does Switching to Muon Hurt Fine-Tuning of Small Open Models? A Case Study on Qwen2.5-0.5B

Muon, an optimizer that orthogonalises the momentum update of each weight matrix, has become a strong alternative to AdamW for pretraining language models. Whether it should also be used to fine-tune the many open models pretrained with AdamW is less clear, and recent studies disagree: one reports a penalty for switching optimizers on a narrow task, another finds none in broad instruction tuning. Both rely mainly on models pretrained from scratch or on models of 4B parameters and above. I examine the question for a small, heavily trained open model, Qwen2.5-0.5B, fine-tuned with AdamW and with Muon on a narrow task (WikiText-2) and on broad instruction data (a Tulu-3 subset), with a separate learning-rate sweep for each optimizer and three seeds for every final configuration. Switching costs little on the target task: validation perplexity rises from 12.47 to 12.53 in the narrow setting, and the target loss does not get worse in the broad one. Forgetting, measured as the increase in loss on held-out web text, is almost entirely determined by how far fine-tuning moves the weights (r ≥ 0.998 across 38 runs), and at equal target loss Muon forgets 1.7–2.3 times more in the narrow setting. Muon saved 6% of peak GPU memory and ran 9–10% slower. At this scale I find little reason to switch.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-22
DOI
https://doi.org/10.5281/zenodo.24430729
Primary Topic
Topic Modeling
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Does Switching to Muon Hurt Fine-Tuning of Small Open Models? A Case Study on Qwen2.5-0.5B

Farhad Rahimov
Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
preprint

Does Switching to Muon Hurt Fine-Tuning of Small Open Models? A Case Study on Qwen2.5-0.5B

Farhad Rahimov
preprint en

Abstract

Muon, an optimizer that orthogonalises the momentum update of each weight matrix, has become a strong alternative to AdamW for pretraining language models. Whether it should also be used to fine-tune the many open models pretrained with AdamW is less clear, and recent studies disagree: one reports a penalty for switching optimizers on a narrow task, another finds none in broad instruction tuning. Both rely mainly on models pretrained from scratch or on models of 4B parameters and above. I examine the question for a small, heavily trained open model, Qwen2.5-0.5B, fine-tuned with AdamW and with Muon on a narrow task (WikiText-2) and on broad instruction data (a Tulu-3 subset), with a separate learning-rate sweep for each optimizer and three seeds for every final configuration. Switching costs little on the target task: validation perplexity rises from 12.47 to 12.53 in the narrow setting, and the target loss does not get worse in the broad one. Forgetting, measured as the increase in loss on held-out web text, is almost entirely determined by how far fine-tuning moves the weights (r ≥ 0.998 across 38 runs), and at equal target loss Muon forgets 1.7–2.3 times more in the narrow setting. Muon saved 6% of peak GPU memory and ran 9–10% slower. At this scale I find little reason to switch.

Zenodo (CERN European Organization for Nuclear Research)
University of Leeds (GB)
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon