A global Swin-Unet Sentinel-2 surface reflectance-based cloud and cloud shadow detection algorithm for the NASA Harmonized Landsat Sentinel-2 (HLS) dataset

The NASA Harmonized Landsat Sentinel-2 (HLS) data provides global coverage atmospherically corrected surface reflectance with a 30m cloud and cloud shadow mask derived using the Fmask algorithm applied to top-of-atmosphere (TOA) reflectance. In this study we demonstrate, as have other researchers, low Sentinel-2 Fmask performance, and present a solution that applies a deep learning Swin-Unet model to the HLS surface reflectance to provide unambiguously improved cloud and cloud shadow detection. The model was trained and assessed using 30m HLS surface reflectance for the 13 Sentinel-2 bands and corresponding CloudSEN12+ annotations, that define cloud, thin cloud, clear, and cloud shadow, and is the largest publicly available expert annotation set. All the CloudSEN12 annotations with coincident HLS Sentinel-2 data were considered. A total of 8672 globally distributed 5 × 5 km data sets were used, 7362 to train the model, 464 for internal model validation, and 846 to independently assess the classification accuracy. The HLS Sentinel-2 Fmask had F1-scores of 0.832 (cloud), 0.546 (cloud shadow), and 0.873 (clear), and the Swin-Unet model had higher performance with F1-scores of 0.891 (cloud and thin cloud combined), 0.710 (cloud shadow), and 0.923 (clear) despite the use of surface and not TOA reflectance. The Swin-Unet thin cloud class had low accuracy (0.604 F1-score) likely due to atmospheric correction issues and thin cloud variability that are discussed. The comprehensively trained model provides a solution for users who wish to improve the HLS Sentinel-2 cloud and cloud shadow masking using the available HLS Sentinel-2 surface reflectance data. • 30 m per-pixel cloud and cloud shadow detection algorithm for HLS Sentinel-2 data. • Swin-Unet deep learning model; clear, cloud, thin cloud, and shadow classes. • Trained on high-quality expert labeled globally distributed CloudSen12+ annotations. • Unambiguously more accurate than HLS Sentinel-2 Fmask. • Despite the use of HLS surface reflectance and not TOA reflectance.