Evaluating Orthogonal Subspace and Sparse Low-Rank Adaptation Under Model-Directed Autonomous Curricula

Orthogonality- and sparsity-based low-rank adaptation methods such as O-LoRA and LoRI are intended to reduce catastrophic forgetting in language models, yet, to our knowledge, they have only been evaluated on task sequences chosen by a person. Self-editing systems such as SEAL break that assumption: the model writes its own training data and can also decide what to learn next. We build a small, fully logged testbed to ask whether the protection these methods give survives when the model picks the order. Qwen2.5-1.5B-Instruct is edited sequentially on 19 facts from 2024 (38 held-out questions) under five configurations and three seeds, 15 runs and 285 edits in total. Vanilla LoRA finishes at 2.6% accuracy, below the untrained model's 7.9%, and damages facts the base model already partly knew more than any other method (50% to 5.6% on those facts), whereas LoRI-style training leaves them intact and O-LoRA-style training falls in between. The OLoRA-style and LoRI-style variants finish at 12.3–14.0%, about ten points above vanilla; the bootstrap intervals for that gap narrowly exclude zero, though we did not correct for multiple comparisons, so we call the evidence suggestive. The Curriculum Transfer Gap is +0.9 points for O-LoRA and −0.9 points for LoRI, roughly a third of one question, with intervals of about ±8 points, so neither can be told apart from zero. Model-chosen orders were no more topic-clustered than the human order and were biased toward the first-listed candidate. We present this as an exploratory, small-scale result, document where our implementations depart from the original methods, and describe a second protocol with control curricula that is implemented but not yet run.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-19
DOI
https://doi.org/10.5281/zenodo.24362975
Primary Topic
Domain Adaptation and Few-Shot Learning
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Evaluating Orthogonal Subspace and Sparse Low-Rank Adaptation Under Model-Directed Autonomous Curricula

Altamash Shaikh, Shiwani Gupta, Anand Maha
Zenodo (CERN European Organization for Nuclear Research)
Domain Adaptation and Few-Shot Learning
preprint

Evaluating Orthogonal Subspace and Sparse Low-Rank Adaptation Under Model-Directed Autonomous Curricula

Altamash Shaikh, Shiwani Gupta, Anand Maha
preprint en

Abstract

Orthogonality- and sparsity-based low-rank adaptation methods such as O-LoRA and LoRI are intended to reduce catastrophic forgetting in language models, yet, to our knowledge, they have only been evaluated on task sequences chosen by a person. Self-editing systems such as SEAL break that assumption: the model writes its own training data and can also decide what to learn next. We build a small, fully logged testbed to ask whether the protection these methods give survives when the model picks the order. Qwen2.5-1.5B-Instruct is edited sequentially on 19 facts from 2024 (38 held-out questions) under five configurations and three seeds, 15 runs and 285 edits in total. Vanilla LoRA finishes at 2.6% accuracy, below the untrained model's 7.9%, and damages facts the base model already partly knew more than any other method (50% to 5.6% on those facts), whereas LoRI-style training leaves them intact and O-LoRA-style training falls in between. The OLoRA-style and LoRI-style variants finish at 12.3–14.0%, about ten points above vanilla; the bootstrap intervals for that gap narrowly exclude zero, though we did not correct for multiple comparisons, so we call the evidence suggestive. The Curriculum Transfer Gap is +0.9 points for O-LoRA and −0.9 points for LoRI, roughly a third of one question, with intervals of about ±8 points, so neither can be told apart from zero. Model-chosen orders were no more topic-clustered than the human order and were biased toward the first-listed candidate. We present this as an exploratory, small-scale result, document where our implementations depart from the original methods, and describe a second protocol with control curricula that is implemented but not yet run.

Zenodo (CERN European Organization for Nuclear Research)
Thakur College of Engineering and Technology (IN)
Domain Adaptation and Few-Shot Learning
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon