Flexible Memory Rotation (FMR): Rotated Representation with Dynamic Regularization to Overcome Catastrophic Forgetting in Continual Knowledge Graph Learning

As a special type of Knowledge Graph (KG), Continual Knowledge Graph Learning (CKGL) plays a pivotal role in various areas such as recommendation systems, search engines, and personalized services, where knowledge dynamically evolves. A challenging problem in CKGL is catastrophic forgetting, where models forget previously learned knowledge upon being trained on new data. To overcome the challenge, this study proposes Flexible Memory Rotation (FMR), a dual-level regularization technique that focuses on both parameter level and structural level. Our idea is inspired by the natural human learning process, which tends to memorize correctly learned knowledge, leverage the learned to acquire new knowledge, and refine incorrectly learned knowledge with newly obtained information. Commonly, existing regularization-based methods fail to mimic this human nature by having a fixed constraint strategy for all model parameters. To this end, the proposed FMR offers flexible constraints based on qualities of learned knowledge evaluated by the Fisher Information Matrix (FIM). Additionally, we identified a limitation of FIM in CKGL, which is the assumption of independence of time steps does not always hold. To overcome this, FMR rotates the parameter space to diagonalize the FIM. This work has four major contributions: 1) develop a novel regularization technique, FMR, a flexible regularization technique, 2) reveal the unexpected failure of FIM in CKGL and provides an easy remedy via parameter space rotation, 3) the comparison experiments on four benchmark datasets designed for CKGL demonstrates improvement over various state-of-the-art (SOTA) CKGL models, and 4) a comprehensive ablation study investigates each component of the proposed model. The source code is available at https://github.com/lijingzhu1/FMR.