Does Switching to Muon Hurt Fine-Tuning of Small Open Models? A Case Study on Qwen2.5-0.5B
Muon, an optimizer that orthogonalises the momentum update of each weight matrix, has become a strong alternative to AdamW for pretraining language models. Whether it should also be used to fine-tune the many open models pretrained with AdamW is less clear, and recent studies disagree: one reports a penalty for switching optimizers on a narrow task, another finds none in broad instruction tuning. Both rely mainly on models pretrained from scratch or on models of 4B parameters and above. I examine the question for a small, heavily trained open model, Qwen2.5-0.5B, fine-tuned with AdamW and with Muon on a narrow task (WikiText-2) and on broad instruction data (a Tulu-3 subset), with a separate learning-rate sweep for each optimizer and three seeds for every final configuration. Switching costs little on the target task: validation perplexity rises from 12.47 to 12.53 in the narrow setting, and the target loss does not get worse in the broad one. Forgetting, measured as the increase in loss on held-out web text, is almost entirely determined by how far fine-tuning moves the weights (r ≥ 0.998 across 38 runs), and at equal target loss Muon forgets 1.7–2.3 times more in the narrow setting. Muon saved 6% of peak GPU memory and ran 9–10% slower. At this scale I find little reason to switch.
Authors
- Farhad Rahimov
Institutions
- University of Leeds (GB)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-22
- DOI
- https://doi.org/10.5281/zenodo.24430729
- Primary Topic
- Topic Modeling
- Type
- preprint