Generative Replay and Multi-steps Knowledge Distillation in Class-Incremental Learning

Data-free generative modeling (GM) methods have gained much attention because pseudo data avoids security and privacy issues, and no amount limitation for storing real data in Class-Incremental Learning (CIL). However, GM-based methods have three drawbacks: un stable quality of generated image, semantic sifting, and domain bias for knowledge and distillation. To tackle with these problems, we propose an effective two-stage training method based on generative replay distillation, namely GA-RCT in this paper. It mainly consists of three modules. The Generative Adaptive Replay (GAR) module in stage one obtainsa well-trained generator for every task, then freezes generator for generating stable images, and trains feature extractor with mixture of real data and generated data. The Global Average Class (GAC) module is designed to reserve class prototypes to reduce semantic drift in the training stage. And the Generative Auxiliary Teaching (GAT) module in stage two performs generative distillation in dual-teacher to student way, for solving domain bias. Experiments on four datasets have shown that GA-RCT performs better than other generative replay methods.