Continual Learning Based on Multi-Teacher Knowledge Distillation

Continual learning aims to enable models to progressively acquire knowledge from novel classes while preserving existing knowledge without catastrophic forgetting of old classes. To achieve this objective, continual learning must operate under three critical constraints: limited training samples per new class, consistent computational costs for learning new classes, and linear memory growth proportional to the number of known classes. To address these constraints, this paper proposes DM-FSCIL (Distillation-MLP Few-Shot Class-Incremental Learning), an architecture comprising three auxiliary teacher networks, one trainable student network, and a dynamically expandable memory. The framework initially trains the student model on abundant base-class data. During incremental learning phases, it employs a multi-teacher knowledge distillation approach to train the network while storing encountered samples of all known classes. Subsequently, the system utilizes a multi-layer perceptron (MLP) module to aggregate base model predictions for student network calibration, effectively mitigating catastrophic forgetting. Comprehensive experiments conducted on Mini-ImageNet, CIFAR100, and CUB200 datasets demonstrate the framework's superior performance in few-shot continual learning scenarios.