Training a Dynamic Growing Mixture Model for Lifelong Learning

Lifelong learning (LLL) defines a training paradigm that aims to continuously acquire and capture new concepts from a sequence of tasks without forgetting. Recently, dynamic expansion models (DEMs) have been proposed to address catastrophic forgetting under the LLL paradigm. However, the efficiency of DEMs lacks a thorough explanation based on theoretical analysis. In this article, we develop a new theoretical framework that interprets the forgetting process of the DEM as increasing the statistical discrepancy distance between the distribution of the probabilistic representation of the new data and the previously learned knowledge. The theoretical analysis shows that adding new components to a mixture model represents a trade-off between model complexity and its performance. Inspired by the theoretical analysis, we introduce a new DEM, called the growing mixture model (GMM), where generative data components are added according to the novelty of the incoming task information compared to what is already known. A new component selection mechanism considering the model's already acquired knowledge is employed for updating new DEM's components, promoting efficient future task learning. We also train a compact student model with samples drawn through the generative mechanisms of the GMM, aiming to accumulate cross-domain representations over time. By employing the student model, we can significantly reduce the number of parameters and make quick inferences during the testing phase.