A Simple but Powerful Baseline for Continual Learning

The application of pre-trained models in continual learning has become the norm, with most methods relying on them. Many approaches freeze or fine-tune the backbone network during training to combat catastrophic forgetting. However, we believe this underutilizes the knowledge embedded in pre-trained models. In particular, recent works have achieved remarkable results by adding the simple Nearest Mean Classifier (NMC) to pre-trained models, demonstrating their inherent strength. This paper introduces a simple but powerful baseline using multi-scale feature fusion and feature clustering to further harness the pre-trained model’s potential. Our method is parameter-free, task-free, and replay-free, fitting both class and domain incremental learning. Results on Split-CIFAR100 and CoRe50 benchmarks for class and domain incremental learning respectively were $86.94 \%$ and $\mathbf{9 0. 5 1 \%}$, surpassing existing methods despite our simple approach. We encourage future work to leverage pre-trained knowledge more deeply rather than just combatting forgetting.