Mitigating Class Confusion in Class-Incremental Semantic Segmentation

Class-incremental semantic segmentation (CSS) aims to continuously classify every pixel in an image to its corresponding semantic class. Existing CSS models suffer from two challenges: catastrophic forgetting and background shift. Moreover, when new classes are introduced over time, they struggle to differentiate between new and old classes that are visually similar. We refer to this problem as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">class confusion</i>. To address the above issues, we propose a novel CSS approach to mitigate class confusion, called MC-Seg. First, we introduce a graph attention decoder (GAD) to capture semantic relationships between embeddings, enabling clearer segmentation across classes. GAD uses joint embeddings as nodes in a graph, enabling visual embeddings to incorporate class information. Furthermore, we propose similarity-aware class discrimination (SCD) to prevent old knowledge from being overwritten by new knowledge, particularly when the two are visually similar. Since visually similar classes are more likely to be confused with each other, SCD increases the distances between their embeddings to preserve essential distinctions. Moreover, to retain the knowledge of other, non-similar classes, we maintain the distances between their embeddings. Extensive experiments on PASCAL VOC 2012, ADE20K and PASCAL Context show that MC-Seg outperforms state-of-the-art CSS methods, especially in alleviating class confusion in incremental learning.