Evaluating Orthogonal Subspace and Sparse Low-Rank Adaptation Under Model-Directed Autonomous Curricula
Orthogonality- and sparsity-based low-rank adaptation methods such as O-LoRA and LoRI are intended to reduce catastrophic forgetting in language models, yet, to our knowledge, they have only been evaluated on task sequences chosen by a person. Self-editing systems such as SEAL break that assumption: the model writes its own training data and can also decide what to learn next. We build a small, fully logged testbed to ask whether the protection these methods give survives when the model picks the order. Qwen2.5-1.5B-Instruct is edited sequentially on 19 facts from 2024 (38 held-out questions) under five configurations and three seeds, 15 runs and 285 edits in total. Vanilla LoRA finishes at 2.6% accuracy, below the untrained model's 7.9%, and damages facts the base model already partly knew more than any other method (50% to 5.6% on those facts), whereas LoRI-style training leaves them intact and O-LoRA-style training falls in between. The OLoRA-style and LoRI-style variants finish at 12.3–14.0%, about ten points above vanilla; the bootstrap intervals for that gap narrowly exclude zero, though we did not correct for multiple comparisons, so we call the evidence suggestive. The Curriculum Transfer Gap is +0.9 points for O-LoRA and −0.9 points for LoRI, roughly a third of one question, with intervals of about ±8 points, so neither can be told apart from zero. Model-chosen orders were no more topic-clustered than the human order and were biased toward the first-listed candidate. We present this as an exploratory, small-scale result, document where our implementations depart from the original methods, and describe a second protocol with control curricula that is implemented but not yet run.
Authors
- Altamash Shaikh
- Shiwani Gupta
- Anand Maha
Institutions
- Thakur College of Engineering and Technology (IN)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-19
- DOI
- https://doi.org/10.5281/zenodo.24362975
- Primary Topic
- Domain Adaptation and Few-Shot Learning
- Type
- preprint