Kalman contrastive unsupervised representation learning

We first propose a Kalman contrastive (KalCo) framework for unsupervised representation learning by dictionary lookup. It builds a dynamic dictionary of encoded representation keys with a queue and a Kalman filter encoder, to which the encoded queries are matched. The large and consistent dictionaries built this way increase the accuracy of KalCo to values much higher than those of the famous momentum contrastive (MoCo) unsupervised learning, which is actually a very simplified version of KalCo with only a fixed scaler momentum coefficient. For a standard pretext task of instance discrimination on the ImageNet-1M (IN-1M) dataset; e.g., KalCo yields an accuracy of 80%, compared to 55% for MoCo. Similar results are obtained also on Instagram-1B (IG-1B). For the same task on a bunch of OpenfMRI datasets, the accuracy is 84%. We then upgrade KalCo to KalCo v2 by using an MLP projection head and more data augmentation, along also with a larger memory bank. The accuracy of KalCo v2 is around the even more impressive amounts of 90% on IN-1M and IG-1B, and 95% on OpenfMRI, the first being about 3% higher than those of three most-cited recent alternatives.