Less is More: Revisiting the Role of Batch Normalization in Test-time Adaptation

This Continual Test-Time Adaptation (CTTA) enables pre-trained models to adapt to evolving data distributions without the need of source data for retraining. Current methods often involve resource-intensive updates or full model modifications, which can lead to error accumulation and catastrophic forgetting. We propose LessBN, a simple yet effective approach that selectively adjusts shallow Batch Normalization (BN) layers at test time, minimizing prediction entropy to boost model confidence. This adjustment preserves deep feature knowledge while enhancing efficiency and mitigating forgetting. The benchmark experiments on CIFAR10C and CIFAR100C demonstrate that LessBN matches or surpasses existing methods with significantly faster adaptation speed, validating its practicality and effectiveness. It can be easily plugged into off-the-shelf models due to its simplicity and generalizability.