Never Forget What You Learn: FK-LoRA for Continual Learning Without Forgetting

When a language model learns new tasks using the adapters, it forgets , because new adapters change how the model computes on old inputs before learning them. FK-LoRA gives every task its own frozen set of adapters and it sends every input to only one of them, so the answer changes only when the route of that input changes. Our main rule, FinderLoRA, does not read the instruction at all , it works as a recognizer on the frozen states of the model, it doesn't depend on the order in which the tasks arrive and it doesn't block any task. It also comes with an audit, which uses held-out inputs to estimate and bound how many of the answers change, and with a variant called KeeperLoRA, which bounds in advance how much a later task can take. When there are no instructions, any method will lose on two tasks together at least the share of inputs that both tasks have in common but with different right answers. The benchmarks hide this , because all of their inputs begin with the instruction. When the instruction is removed, the two published methods that we ran, GainLoRA and InfLoRA, change 414 and 746 out of 1,200 answers (65 and 370 when the instruction is kept). This loss was not forced, since no input in these benchmarks occurs in two tasks, and the recognizer sends 99.0% of the test inputs to their own task on 15 development tasks, 97.8% on a second benchmark and 94.6% on 30 unseen tasks. On five tasks (three seeds, a tenth of the published epochs), the plain adapters behind it change 18 answers and end at 77.2 Rouge-L, against 51.0 and 38.9 for the other two methods , and most of that gap comes from learning. With the published 100 epochs (one seed), the two methods learn the tasks almost as well as the plain adapters trained with the same epochs, but then they forget 25 and 66 points and end at 56.4 and 25.4, against 80.2 for the plain adapters behind it (77.5 at 10 epochs, same seed), and when their instructions are kept they forget only 0.2 and 3.5 points. When the inputs carry an instruction, a second rule gives each task a zone around the word-vectors of its instruction, and we prove that an answer that falls inside a zone never changes , and no answer to an earlier task changed (0 of 1,200 on five tasks, three seeds; 0 of 1,097 on fifteen). When the instructions are reworded, they fall outside the zones , on machine paraphrases of 700 real tasks the zones certify 39.8% to 50.4% of the inputs for the right task, and the five attempts we made to widen them all missed their bars. Since the recognizer ignores the instruction, no rewording can change its routes , with instructions (five tasks, three seeds, development rewordings) it ends at 77.9 under a heavy rewording where the zones end at 74.3, but with the original instructions the zones end higher (79.8 against 79.1) and change no answer, while the recognizer changes 9 of 1,200. When there is no instruction, the inputs that occur in two tasks can at best be split between them , the bound in advance is loose , all of the runs use T5-base, and none of it is in the published setting. The stand-alone code is also released.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-20
DOI
https://doi.org/10.5281/zenodo.24396121
Primary Topic
Domain Adaptation and Few-Shot Learning
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Never Forget What You Learn: FK-LoRA for Continual Learning Without Forgetting

Abhishek Jangid, Shivkumar Bammidi
1 links
Zenodo (CERN European Organization for Nuclear Research)
Domain Adaptation and Few-Shot Learning
preprint

Never Forget What You Learn: FK-LoRA for Continual Learning Without Forgetting

Abhishek Jangid, Shivkumar Bammidi
preprint en

Abstract

When a language model learns new tasks using the adapters, it forgets , because new adapters change how the model computes on old inputs before learning them. FK-LoRA gives every task its own frozen set of adapters and it sends every input to only one of them, so the answer changes only when the route of that input changes. Our main rule, FinderLoRA, does not read the instruction at all , it works as a recognizer on the frozen states of the model, it doesn't depend on the order in which the tasks arrive and it doesn't block any task. It also comes with an audit, which uses held-out inputs to estimate and bound how many of the answers change, and with a variant called KeeperLoRA, which bounds in advance how much a later task can take. When there are no instructions, any method will lose on two tasks together at least the share of inputs that both tasks have in common but with different right answers. The benchmarks hide this , because all of their inputs begin with the instruction. When the instruction is removed, the two published methods that we ran, GainLoRA and InfLoRA, change 414 and 746 out of 1,200 answers (65 and 370 when the instruction is kept). This loss was not forced, since no input in these benchmarks occurs in two tasks, and the recognizer sends 99.0% of the test inputs to their own task on 15 development tasks, 97.8% on a second benchmark and 94.6% on 30 unseen tasks. On five tasks (three seeds, a tenth of the published epochs), the plain adapters behind it change 18 answers and end at 77.2 Rouge-L, against 51.0 and 38.9 for the other two methods , and most of that gap comes from learning. With the published 100 epochs (one seed), the two methods learn the tasks almost as well as the plain adapters trained with the same epochs, but then they forget 25 and 66 points and end at 56.4 and 25.4, against 80.2 for the plain adapters behind it (77.5 at 10 epochs, same seed), and when their instructions are kept they forget only 0.2 and 3.5 points. When the inputs carry an instruction, a second rule gives each task a zone around the word-vectors of its instruction, and we prove that an answer that falls inside a zone never changes , and no answer to an earlier task changed (0 of 1,200 on five tasks, three seeds; 0 of 1,097 on fifteen). When the instructions are reworded, they fall outside the zones , on machine paraphrases of 700 real tasks the zones certify 39.8% to 50.4% of the inputs for the right task, and the five attempts we made to widen them all missed their bars. Since the recognizer ignores the instruction, no rewording can change its routes , with instructions (five tasks, three seeds, development rewordings) it ends at 77.9 under a heavy rewording where the zones end at 74.3, but with the original instructions the zones end higher (79.8 against 79.1) and change no answer, while the recognizer changes 9 of 1,200. When there is no instruction, the inputs that occur in two tasks can at best be split between them , the bound in advance is loose , all of the runs use T5-base, and none of it is in the published setting. The stand-alone code is also released.

Zenodo (CERN European Organization for Nuclear Research)
Domain Adaptation and Few-Shot Learning
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon