Resilient Test-Time Adaptation by Mitigating Batch-Normalization Overfitting

Test-time domain adaptation adjusts a source domain model to accommodate previously unseen domain shifts in a target domain during inference. In real-world scenarios, domain shifts continually evolve, and test data are often non-independent and identically distributed (non-i.i.d.). Existing methods update batch normalization (BN) statistics (mean and variance) based on test batch statistics to mitigate domain shifts and use a memory bank to provide approximate i.i.d. sampling by selectively storing samples. However, excessive updates to BN statistics lead to overfitting to specific domain shifts. To address this issue, we propose a resilient practical test-time adaptation method (ResiTTA), employing soft constraints on the BN statistics and a low-entropy sampling strategy, which reduces overfitting on domain shifts and enables rapid adaptation. Specifically, we develop a resilient batch normalization (BN) with estimated statistics and soft constraints between the source and the estimated statistics. The soft constraints regularize the estimated statistics to mitigate overfitting caused by the excessive updates. To avoid overfitting, we design a low-entropy memory bank that accounts for sample uncertainty and class balance. We adapt the source domain model via a teacher-student self-training adaptation on the samples from the memory, incorporating the soft constraints’ updates to BN. Our ResiTTA obtains state-of-the-art results on various benchmarks. We release our code<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>.