A Dynamic Class Incremental Learning Expansion Architecture Based on Multi-Level Deep Knowledge Distillation

The dynamic expansion architectures are becoming increasingly popular in class incremental learning because of its significant advantages in overcoming catastrophic forgetting. However, most of the papers on these architectures ignore the following problems: (1) because of the structure of the dynamic expansion architecture itself, there are three kinds of confusion problems, i.e., the model is biased towards new samples (old-new confusion, ONC), the differences between samples of similar classes in the same task are not well learned (within-task confusion, WTC), and the differences between different tasks are not well learned (inter-task confusion, ITC); (2) due to the large number of samples and the varying training difficulty for each sample, we should include the consideration of the classification difficulty of different samples when designing the incremental learning model. To this regard, we propose a class incremental learning Dynamic Expansion Architecture based on Multi-Level deep knowledge Distillation (MLD-DEA), which takes the knowledge by multi-level deep knowledge distillation. In addition, we also introduced the samples' entropy value to measure the samples' uncertainty, which is used as an index to evaluate the difficulty of each sample. Finally, we introduce an efficient attention mechanism to balance the feature fusion of old and new class samples. We conduct experiments on the CIFAR100 and ImageNet100 datasets. The experimental results show that the accuracies of the proposed algorithm are significantly improved compared to existing state-of-the-art methods on the baseline datasets. The superiority remains good under different memory pools.