Adapt & Align: Continual Learning with Generative Models’ Latent Space Alignment

Neural networks suffer from abrupt loss in performance when retrained with additional data from different distributions. At the same time, training with additional data without access to the previous examples rarely improves the model’s performance. We propose Adapt & Align, a novel continual learning framework that leverages generative models to align their latent representations across tasks. The approach is divided into two phases: • Local Training: Train a generative model (e.g., a Variational Autoencoder (VAE) or a Generative Adversarial Network (GAN)) on the current task to capture task-specific features. • Global Training: Use a translator network to map these task-specific latent representations into a unified global latent space, thereby facilitating both forward and backward knowledge transfer. Experiments on benchmark datasets (e.g., MNIST, Omniglot, CIFAR, CelebA) as well as real-world application for particle simulation at CERN demonstrate that Adapt & Align mitigates catastrophic forgetting and improves generation quality as indicated by metrics such as Fréchet Inception Distance (FID), distribution precision and recall, or accuracy for the downstream classification task. Ablation studies confirm the critical role of each component. • New method for continual learning based on the alignment of representations in generative models. • Evaluated on generative task with VAEs and GANs. • Extension to downstream – classification task.