Evaluating Neural Network Parameter Obliteration in Catastrophic Forgetting

One of the major weaknesses of Neural Networks is their inability to learn multiple functions (tasks) sequentially. This phenomenon is observed when a data distribution shift is encountered in continual learning, leading to catastrophic forgetting. Such ability is foremost important to develop models with general intelligence. In this paper, we evaluate how different parts of the neural network contribute to overall forgetting by investigating the training dynamics during different methods of continual learning. We evaluate and analyze the learning dynamics in catastrophic forgetting on frequently used classification models on CIFAR-10 and Fashion-MNIST datasets. Our findings reveal a consistent trend in continual learning scenarios, both in domain-incremental and class-incremental settings. Specifically, gamma means demonstrate stability in earlier layers but exhibit increased plasticity in later layers, indicating heightened adaptability to changes in data distributions.