PLANCKIAN JITTER: COUNTERING THE COLOR- CRIPPLING EFFECTS OF COLOR JITTER ON SELF- SUPERVISED TRAINING

Several recent works on self-supervised learning are trained by mapping differentaugmentations of the same image to the same feature representation. The dataaugmentations used are of crucial importance to the quality of learned feature representations. In this paper, we analyze how the color jitter traditionally used indata augmentation negatively impacts the quality of the color features in learnedfeature representations. To address this problem, we propose a more realistic,physics-based color data augmentation – which we call Planckian Jitter – thatcreates realistic variations in chromaticity and produces a model robust to illumination changes that can be commonly observed in real life, while maintainingthe ability to discriminate image content based on color information. Experimentsconfirm that such a representation is complementary to the representations learnedwith the currently-used color jitter augmentation and that a simple concatenationleads to significant performance gains on a wide range of downstream datasets.In addition, we present a color sensitivity analysis that documents the impact ofdifferent training methods on model neurons and shows that the performance ofthe learned features is robust with respect to illuminant variations. Official codeavailable at: https://github.com/TheZino/PlanckianJitter