Variance-Covariance Regularization Improves Continual Learning

In this work, we explore the benefits of Variance-Covariance Regularization in Continual Learning (CL). Neural networks suffer from abrupt performance loss when updated with additional data. Numerous CL approaches try to mitigate this problem by preserving the already accumulated knowledge within the network. We propose to look at this problem from a different perspective and analyze how we can prepare the continually trained model for future changes. With a series of initial experiments, we show that increasing the diversity of extracted features can improve the classifier’s robustness, enhancing its immunity to future changes. Based on this observation, we introduce Variance-Covariance Regularization in Continual Learning. As we explore previously ignored aspects of CL, our method is orthogonal to most of the CL techniques. It can be combined with them, yielding a significant performance gain across different datasets, training scenarios, and architectures.