Deciphering Barlow Twins: Reduncy Reduction is Insufficient and Normalization is Key

Barlow Twins is a feature-contrastive self-supervised learning framework built on the principle of redundancy reduction.The idea is to train a network by maximizing the correlation between corresponding features and minimizing the correlation between non-corresponding features in distorted views of the same image, through this facilitating effective pretraining of a backbone network for a subsequent classification head.This is achieved by diagonalizing the cross-correlation matrix of the network's representations and scaling it towards the identity matrix.We show that the cross-correlation matrix of distorted images is inherently symmetric, independent of the backbone network's weights, which leads to two key insights: (i) the cross-correlation matrix can always be diagonalized using a linear transformation (layer), and (ii) the core idea of maximizing correlations between corresponding features while minimizing them for non-corresponding features alone is insufficient for effective backbone network pretraining.Nevertheless, Barlow Twins provide highly effective pretraining.We show that this is due to the normalization of the cross-correlation matrix in the Barlow Twins cost function.This normalization leads to minima of the cost function which are equivalent to the minima of sample contrastive approaches to enforce invariance.