Improving Shift Invariance in Convolutional Neural Networks with Translation Invariant Polyphase Sampling

Convolutional neural networks (CNNs), widely deployed in several applications, contain downsampling operators in their pooling layers which have been observed to be sensitive to pixel-level shift, affecting the robustness of CNNs. We study shift invariance through the lens of maximum-sampling bias (MSB) and find MSB to be negatively cor-related with shift invariance. Based on this insight, we propose a learnable pooling operator called Translation Invariant Polyphase Sampling (TIPS) to reduce MSB and learn translation-invariant representations. TIPS results in consistent performance gains on multiple benchmarks for image classification, object detection, and semantic segmentation in terms of accuracy, shift consistency, shift fi-delity, as well as improvements in adversarial and distributional robustness. TIPS results in the lowest MSB compared to all previous methods, thus explaining the strong empirical results. TIPS can be integrated into any CNN and can be trained end-to-end with marginal computational overhead. Code: https://github.com/sourajitcs/tips/