Feature Transformation for Learning with Selective Forgetting

Continual learning aims to learn new tasks while maintaining the ability to solve past tasks. However, retaining all knowledge learned in the past may pose practical risks, such as data leakage. A recent study [1] addresses this problem by introducing a new continual learning paradigm called Learning with Selective Forgetting, allowing the selection of the classes to be forgotten at each continual learning stage. The existing approach is to embed a class-specific pattern called mnemonic code into the input image; this serves to “bind” the images within the same class in the feature space, allowing for sharp control of memorization/forgetting of each class by simply keeping/discarding the code. One limitation of this approach, however, is the inconsistency that the code is embedded in the input image, even though the control of memorization/forgetting relies on the binding effect in the feature space. In this paper, we propose a novel approach that directly embeds the codes into the features. This idea brings a new technical challenge. Unlike in the case of the existing approach, where the code is embedded into the input image whose scale is known and bounded, i.e., typically 0-255, the scale of the intermediate features is generally unknown and unbounded, making it impossible to match the scale of the code to be embedded. We propose a new normalization layer, named StandMix, which simultaneously aligns the scale and embeds the code into the intermediate feature. We also introduce a code optimization approach to further enhance its forgetting performance. Experimental results demonstrate that the proposed method outperforms the existing approach and alternative normalization layers.