Multi-Domain Incremental Learning for Semantic Segmentation via Visual Domain Prompt in Remote Sensing Data

Domain incremental learning for semantic segmentation has gained lots of attention due to its importance for many fields including urban planning and autonomous driving. The catastrophic forgetting problem caused by domain shift has been alleviated by structure expansion of the model or data rehearsal. However, these methods ignore similar contextual knowledge between the new and the old data domain and assume that new knowledge and old knowledge are completely mutually exclusive, which cause the model to be trained in a suboptimal direction. Motivated by the prompt learning, we proposed a new domain incremental learning framework named RS-VDP. The key innovation of RS-VDP is to utilize a visual domain prompt to change the optimization direction from input data space and feature space. First, we designed a domain prompt based on a dynamic location module, which applied a visual domain prompt according to a local entropy map to update the distribution of the input images. Second, in order to filter the feature vectors with high confidence, a representation feature alignment based on an entropy map module is proposed. This module ensures the accuracy and stability of the feature vectors involved in the regularization loss, alleviating the problem of semantic drift. Finally, we introduced a new evaluation metric to measure the overall performance of the incremental learning models, solving the problem that the traditional evaluation metric is affected by the single-task accuracy. Comprehensive experiments demonstrated the effectiveness of the proposed method by significantly reducing the degree of catastrophic forgetting.