CSEA: Confidence-guided Semantic-Enhanced Alignment for Online Test-Time Adaptation

Abstract Online test-time adaptation (OTTA) is a more general method for accomplishing domain adaptation in real-world scenarios because it just needs a pre-trained source model to adapt the mini-batch target online. One popular approach is to adapt the model with empirical risk minimization using generated pseudo-labels. However, it can fail to learn optimal feature space due to the noisy pseudo-labels under domain shift. To address this restriction, we propose a novel online test-time adaptation technique called Confidence-guided Semantic-Enhanced Alignment (CSEA) that separates the mini-batch target into distinct subsets based on the pseudo-labels confidence and applies tailored constraints to each subset for reliable target structure mining. We first divide the mini-batch target data into confident and non-confident based on an online updated memory bank for each class. Then, we customize the alignment strategy to fit each subset best. Specifically, we propose a Re-weighted Local Clustering for the confident subset to learn the local semantic structure information of unlabeled target. Further, we propose a Semantic-enhanced Alignment that applied to the non-confident subset to guide the model to learn additional discriminative semantic knowledge. Extensive experiments on three OTTA benchmarks indicate effectiveness of the proposed CSEA.