Self-supervised representation learning for cloud detection using Sentinel-2 images

The unavoidable presence of clouds and their shadows in optical satellite imagery hinders the true spectral response of the Earth’s underlying surface. Accurate cloud and cloud shadow detection is therefore a crucial preprocessing step for optical satellite images and any downstream analysis. Various methods have been developed to address this critical task and can be broadly categorized into physical rule-based methods and learning based methods. In recent years, machine learning based methods, particularly deep learning frameworks, have proven to outperform physical rule-based models. However, these approaches are mostly fully supervised and require a large amount of pixel-level annotations whose acquisition is costly and time consuming. In this work, we propose to address cloud and cloud shadow detection in optical satellite images using self-supervised representation learning, a machine learning paradigm that focuses on extracting relevant representations from unlabeled data, which can then be used as an effective starting point to fine-tune models with few labeled data in a supervised fashion. These approaches have been shown to perform competitively with fully supervised methods without the requirement of large annotation datasets. Specifically, we assessed two self-supervised representation learning methods that use different philosophies about self-supervision: Momentum Contrast (MoCo), based on contrastive learning and DeepCluster, based on clustering. Using two publicly available Sentinel-2 cloud datasets, namely WHUS2–CD+ and CloudSEN12, we show that MoCo and DeepCluster, trained with only 25 % of the annotated data, can perform better than physical rule-based methods such as FMask and Sen2Cor, weakly supervised methods and even several fully supervised methods. These results highlight the strong applicability of self-supervised representation learning methods to the task of cloud and cloud shadow detection with self-supervised pretraining leading to fine-tuned models that outperform industry standards and achieve near state-of-the-art performance with a fraction of the data. • Two self-supervised approaches are leveraged for cloud and cloud shadow detection. • MoCo and DeepCluster outperformed industry standards such as FMask and Sen2Cor with few labels. • Reliable performance was obtained for both methods using 25 % of annotations in the training sets. • MoCo and DeepCluster handle clouds better than cloud shadows.