Selected Replay with Distillation for Continual Semantic Segmentation

Although deep neural networks have demonstrated exceptional performance in various fields, especially in image processing, they still face some significant unresolved challenges. Catastrophic forgetting is one of the typical challenges commonly encountered in deep learning models. This issue becomes apparent when new training data introduces new categories to adapt to environmental changes, while the old training data is no longer available. As a result, the model tends to forget previously learned knowledge. In recent years, a variety of solutions and incremental learning frameworks have been proposed to address this issue, particularly showing effectiveness in classification and detection tasks. In this paper, we extend the problem to semantic segmentation, which involves pixel-level classification. We conducted an in-depth analysis of the root causes of catastrophic forgetting and identified two key issues arising from the unavailability of old data samples during the new training phase. First, previously learned knowledge is often mistakenly labeled as background in the new training phase, causing a “background shift” phenomenon. Second, the data distribution of old and new classes becomes highly imbalanced. To tackle these issues, we propose a strategy that combines replay and knowledge distillation to effectively alleviate catastrophic forgetting.