Wake-Sleep Null-Space Consolidation: Quantifying the Stability-Plasticity Frontier in Sequential LLM Adaptation

Official empirical benchmark and research release for Wake-Sleep Null-Space Consolidation (WS-NSC), an algebraic continual learning framework designed for autoregressive Large Language Models (LLMs). Abstract: Sequential adaptation of Transformers without raw data rehearsal induces catastrophic forgetting. While Parameter-Efficient Fine-Tuning (PEFT) minimizes memory footprints, merging learned low-rank updates directly into base weights corrupts pre-existing representations. We empirically study Wake-Sleep Null-Space Consolidation (WS-NSC), an algebraic consolidation mechanism that projects low-rank updates onto the empirical orthogonal complement of historical activation covariances. Evaluating SmolLM-135M on a sequential stream (GSM8K → IMDb → AG News) anchored by WikiText-2, we demonstrate that WS-NSC eliminates catastrophic forgetting (-0.39% relative drift vs. +5.83% in direct merge) and halts anchor drift. However, this stability incurs an acute plasticity penalty, reducing retained task gains to 37.7% (ρ=0.90) and 24.4% (ρ=0.95). We show that retaining parameter update norm (~74%) does not guarantee functional retention (~38%), formalizing the empirical trade-off of hard null-space projection in nonlinear Transformers. Code & Benchmark Repository: https://github.com/AkshatRaj00/ws-nsc