Disentangle Source and Target Knowledge for Continual Test-Time Adaptation

Continual Test-Time Adaptation (CoTTA) task is proposed to tackle the challenges of constant domain shifts during testing. The goals are twofold: 1) to preserve the knowledge from the source domain without source data and 2) to effectively extract target knowledge using unlabeled target domain data. Existing works primarily focus on either source or target knowledge, attempting to learn both in a mixed manner. This may harm the source knowledge preser-vation and target knowledge extraction. To this end, this pa-per proposes a Source and Target knowledge Disentangle Transformer (SoTa-DiT) with the prompting mechanism. Specifically, in a vision transformer (ViT), we employ source and target prompts, supervised by two groups of deliber-ately designed loss functions, to learn source and target knowledge separately. The source prompt focuses on anti-source-forgetting by extracting and preserving knowledge from the source model, while the target prompt focuses on protarget-extracting using target data contrastive learning. With comprehensive evaluations across various datasets using different ViT backbones, we demonstrate that this dual-prompt architecture of SoTa-DiT is effective and that disentangling knowledge with the prompts benefits CoTTA. As a result, SoTa-DiT significantly improves image classification accuracy under the CoTTA setting.