Ask What Forgetting Would Cost: A Label-Free Judge Gates Nightly Replay of Free-Text Facts

Can a model decide what is worth remembering without anyone telling it what matters? We test this on nightly LoRA consolidation, where whatever is not replayed is forgotten. Our earlier work showed that replaying facts by criticality protects them, but on template data and with a probe trained on human labels. Here we remove both. The facts are 150 free-text sentences in Spanish and English, and the selector uses no labels: a 1.5B model is asked once per fact whether forgetting it could cause harm (health, safety, money, legal), and its probability of “Yes” ranks the facts at AUROC 0.992. Over fifteen nights of LoRA consolidation in Qwen2.5-0.5B-Instruct, judge-gated replay retains 19.6 of 38 critical facts, against 8.8 for random replay and 11.0 for replaying what was forgotten, winning in all five seeds (exact McNemar p = 1.2e-8 and 3.3e-6). The gain is largest on implicit critical facts (10.6 of 18 against 3.2), and a keyword selector marks 12 lexical traps as salient where the judge marks none. A replication with Llama-3.2-1B-Instruct, under a rule written before launch, gives 22.0 against 11.7 with no loss of trivial facts. Knowing that a fact was forgotten adds 2.2 facts over chance; knowing what it was worth adds 10.8. All data are LLM-written, the students are small, and every deviation from the written plan is reported. Data, code and full logs are included.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-12
DOI
https://doi.org/10.5281/zenodo.24164502
Primary Topic
Topic Modeling
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Ask What Forgetting Would Cost: A Label-Free Judge Gates Nightly Replay of Free-Text Facts

Sebastian Goin del Rio
Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
preprint

Ask What Forgetting Would Cost: A Label-Free Judge Gates Nightly Replay of Free-Text Facts

Sebastian Goin del Rio
preprint en

Abstract

Can a model decide what is worth remembering without anyone telling it what matters? We test this on nightly LoRA consolidation, where whatever is not replayed is forgotten. Our earlier work showed that replaying facts by criticality protects them, but on template data and with a probe trained on human labels. Here we remove both. The facts are 150 free-text sentences in Spanish and English, and the selector uses no labels: a 1.5B model is asked once per fact whether forgetting it could cause harm (health, safety, money, legal), and its probability of “Yes” ranks the facts at AUROC 0.992. Over fifteen nights of LoRA consolidation in Qwen2.5-0.5B-Instruct, judge-gated replay retains 19.6 of 38 critical facts, against 8.8 for random replay and 11.0 for replaying what was forgotten, winning in all five seeds (exact McNemar p = 1.2e-8 and 3.3e-6). The gain is largest on implicit critical facts (10.6 of 18 against 3.2), and a keyword selector marks 12 lexical traps as salient where the judge marks none. A replication with Llama-3.2-1B-Instruct, under a rule written before launch, gives 22.0 against 11.7 with no loss of trivial facts. Knowing that a fact was forgotten adds 2.2 facts over chance; knowing what it was worth adds 10.8. All data are LLM-written, the students are small, and every deviation from the written plan is reported. Data, code and full logs are included.

Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon