Adaptive Low-Rank Product Transformers with Dynamic Expert Routing for Online Continual Learning

Inspired by the role of sleep in biological continual learning, we introduce RVW, a transformer architecture for online continual adaptation of pretrained models. RVW maintains a small pool of per-layer experts that grow and are pruned in response to distribution shift, with no replay buffer and no explicit task identifier. Applied to TinyLlama-1.1B on a 15,000-chunk six-domain stream, RVW reaches 40 average held-out PPL, substantially better than EWC (158), fine-tuning (164), and LoRA (448) on the same parameter-matched base, while preserving prior-domain performance. Threshold sweeps suggest that learned knowledge is encoded in the routing combination of experts across layers rather than individual experts specializing on specific domains.