Towards Fully Test-Time Adaptation via Variance Balancing and Semantic Augmentation

Fully test-time adaptation (FTTA) is to adapt a model trained on a source domain to a target domain during the testing phase. Traditional methods like entropy minimization primarily focus on reducing uncertainty in output predictions, yet often overlook the diversity in target prediction results, which is critical for unbalanced classes in complex datasets. To address this, our study introduces a new method named Variance Balancing and Semantic Augmentation (VBSA). VBSA begins by maximizing the sum of singular values in predictions, coupled with a novel variance penalization strategy. This strategy not only focuses the model on unbalanced classes but also mitigates the overfitting risk associated with singular value maximization, thereby ensuring a balanced emphasis across various classes and enhancing the diversity of prediction results. Furthermore, VBSA incorporates semantic data augmentation using data from previous batches, offering semantic-level augmentation for all classes, with particular benefits for unbalanced ones. Extensive experiments demonstrate that our VBSA method has produced the most advanced performance.