The Anchor Accumulation Problem: Bounded Anchor Sets Recover Plasticity in Usage-Localized Elastic Weight Consolidation

Usage-localized elastic weight consolidation (DeLP) appends one anchor set — a parameter snapshot plus a diagonal Fisher estimate — per finished task, so its protection penalty grows without bound in both storage and constraint mass. We treat that anchor stack itself as the object of study. On Permuted-MNIST, bounding it at M entries recovers the learning term of the retention decomposition and lifts final accuracy by +4.51 points at M=5 (T=10, n=10 paired seeds, t(9)=21.4) and +8.98 points at T=20 (t(9)=19.1), rising to +12.3 at M=2 (T=10, t(9)=39.6); the effect replicates on Fashion-MNIST (+13.6, t(9)=18.3) and, more weakly, on Split-CIFAR-100 with a ResNet-18 backbone, where at T=20 five of six bounded policies beat unbounded stacking (up to +5.0, t(9)=7.1; Fisher-cosine merge, which trades a small BWT cost for storage, is the exception at -0.5). Three further results frame the mechanism. (i) The budget and lambda are orthogonal knobs: through per-task accuracy profiles, the budget controls the age structure of forgetting while lambda controls its total amount. (ii) Eviction criteria keyed to an anchor's current constraint mass fail catastrophically — they grandfather stale anchors and evict new ones at birth, driving BWT to -34.5 at T=20 — whereas age-first criteria are immune, and trace instrumentation of the anchor stacks shows the mechanism directly. (iii) Fisher-cosine merging removes exactly the redundant inter-anchor cross terms: the penalty's value drops one-sidedly (43-45% per-run maxima at M=2, worst 64%; 18-34% at M=5) while its gradient is unchanged — merging shifts only a parameter-independent constant, which has no effect on SGD — so the accuracy trajectory stays indistinguishable from unbounded and the memory axis is separated from the plasticity axis. Single-anchor references bound the claims: plain EWC maximizes the learning term at 3-5x the forgetting, and online EWC with a tuned discount overtakes the budget at every matched grid cell, revealing that discounting is itself a soft bound; soft and hard bounds collapse onto one log-linear law relating the learning term to the protected-mass horizon (learning term ~= 84 - 11 ln h, R2=0.99, per-seed slope -10.9+/-0.6, monotone at every grid lambda), so bounding accumulated anchor mass — not the eviction mechanism — is the active ingredient. On Split-CIFAR-100 a no-protection baseline (lambda=0) remains above every protected arm, showing that protection itself must be checked against a no-protection reference before its mechanism is credited. This is the revised version, deposited after the Neurocomputing editorial decision NEUCOM-D-26-24422 to establish a timestamped public record independently of journal acceptance. Relative to the submitted version it (a) corrects the Split-CIFAR-100 protocol description to T contiguous slices of 100/T classes (5 classes per task at T=20, not 10), (b) replaces the blanket 'every bounded policy' with the exact five-of-six result that the table supports, (c) states the merge penalty change as value-level with an unchanged gradient, (d) re-derives every headline number from the raw records, and (e) adds an explicit auditable/ policy-selectable framing as a contribution. Files: main.pdf (full paper, 28 pp.), LaTeX source with both appendix matrices, five figures (PDF+PNG), the experiment code (bounded-anchor grid on Permuted-MNIST/Fashion-MNIST and Split-CIFAR-100, EWC reference arms, analysis and figure scripts, plus the two vendored base modules), the pre-registration and data-closure documents locked before collection, 27 raw per-seed JSON record files, and — new in this version — an audit/ directory containing the nine audit scripts that re-derive roughly 150 quoted numbers from those records, a fix script that regenerates both appendix matrices, a 27-check regression (27 PASS / 0 FAIL), the full revision checklist, and the editorial decision itself. Companion repository: github.com/zhouranxi/delp-anchor-budget . Related records: precursor DeLP audit doi.org/10.5281/zenodo.23602281 ; S1 segment-specialization audit doi.org/10.5281/zenodo.23637973

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-10-26
DOI
https://doi.org/10.5281/zenodo.23668523
Primary Topic
Domain Adaptation and Few-Shot Learning
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

The Anchor Accumulation Problem: Bounded Anchor Sets Recover Plasticity in Usage-Localized Elastic Weight Consolidation

Ranxi Zhou
Zenodo (CERN European Organization for Nuclear Research)
Domain Adaptation and Few-Shot Learning
preprint

The Anchor Accumulation Problem: Bounded Anchor Sets Recover Plasticity in Usage-Localized Elastic Weight Consolidation

Ranxi Zhou
preprint en

Abstract

Usage-localized elastic weight consolidation (DeLP) appends one anchor set — a parameter snapshot plus a diagonal Fisher estimate — per finished task, so its protection penalty grows without bound in both storage and constraint mass. We treat that anchor stack itself as the object of study. On Permuted-MNIST, bounding it at M entries recovers the learning term of the retention decomposition and lifts final accuracy by +4.51 points at M=5 (T=10, n=10 paired seeds, t(9)=21.4) and +8.98 points at T=20 (t(9)=19.1), rising to +12.3 at M=2 (T=10, t(9)=39.6); the effect replicates on Fashion-MNIST (+13.6, t(9)=18.3) and, more weakly, on Split-CIFAR-100 with a ResNet-18 backbone, where at T=20 five of six bounded policies beat unbounded stacking (up to +5.0, t(9)=7.1; Fisher-cosine merge, which trades a small BWT cost for storage, is the exception at -0.5). Three further results frame the mechanism. (i) The budget and lambda are orthogonal knobs: through per-task accuracy profiles, the budget controls the age structure of forgetting while lambda controls its total amount. (ii) Eviction criteria keyed to an anchor's current constraint mass fail catastrophically — they grandfather stale anchors and evict new ones at birth, driving BWT to -34.5 at T=20 — whereas age-first criteria are immune, and trace instrumentation of the anchor stacks shows the mechanism directly. (iii) Fisher-cosine merging removes exactly the redundant inter-anchor cross terms: the penalty's value drops one-sidedly (43-45% per-run maxima at M=2, worst 64%; 18-34% at M=5) while its gradient is unchanged — merging shifts only a parameter-independent constant, which has no effect on SGD — so the accuracy trajectory stays indistinguishable from unbounded and the memory axis is separated from the plasticity axis. Single-anchor references bound the claims: plain EWC maximizes the learning term at 3-5x the forgetting, and online EWC with a tuned discount overtakes the budget at every matched grid cell, revealing that discounting is itself a soft bound; soft and hard bounds collapse onto one log-linear law relating the learning term to the protected-mass horizon (learning term ~= 84 - 11 ln h, R2=0.99, per-seed slope -10.9+/-0.6, monotone at every grid lambda), so bounding accumulated anchor mass — not the eviction mechanism — is the active ingredient. On Split-CIFAR-100 a no-protection baseline (lambda=0) remains above every protected arm, showing that protection itself must be checked against a no-protection reference before its mechanism is credited. This is the revised version, deposited after the Neurocomputing editorial decision NEUCOM-D-26-24422 to establish a timestamped public record independently of journal acceptance. Relative to the submitted version it (a) corrects the Split-CIFAR-100 protocol description to T contiguous slices of 100/T classes (5 classes per task at T=20, not 10), (b) replaces the blanket 'every bounded policy' with the exact five-of-six result that the table supports, (c) states the merge penalty change as value-level with an unchanged gradient, (d) re-derives every headline number from the raw records, and (e) adds an explicit auditable/ policy-selectable framing as a contribution. Files: main.pdf (full paper, 28 pp.), LaTeX source with both appendix matrices, five figures (PDF+PNG), the experiment code (bounded-anchor grid on Permuted-MNIST/Fashion-MNIST and Split-CIFAR-100, EWC reference arms, analysis and figure scripts, plus the two vendored base modules), the pre-registration and data-closure documents locked before collection, 27 raw per-seed JSON record files, and — new in this version — an audit/ directory containing the nine audit scripts that re-derive roughly 150 quoted numbers from those records, a fix script that regenerates both appendix matrices, a 27-check regression (27 PASS / 0 FAIL), the full revision checklist, and the editorial decision itself. Companion repository: github.com/zhouranxi/delp-anchor-budget . Related records: precursor DeLP audit doi.org/10.5281/zenodo.23602281 ; S1 segment-specialization audit doi.org/10.5281/zenodo.23637973

Zenodo (CERN European Organization for Nuclear Research)
Domain Adaptation and Few-Shot Learning
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon