Zero-shot multi-class semantic segmentation of remote sensing images using SAM 2 with prior database information

Abstract. Remote sensing images (RSI) play a central role in the regular update of land cover datasets. A first step in this process is RSI semantic segmentation, which is mainly solved by deep learning today. Vision foundation models (VFM) have gained increasing importance in this context. Having been trained on large datasets, VFM for segmentation can yield good results on data from various domains without further training. We present a new method for using the VFM Segment Anything Model 2 (SAM 2) for multi-class semantic segmentation of Sentinel-2 images that does not require training data. Our method is based on a prompt engineering approach, using SAM 2 in its pre-trained form and generating different prompt types on the basis of existing topographic data. We also propose a post-processing step for merging the output of SAM 2 to obtain a multi-class label image. The results of our experiments show that our method achieves an overall accuracy (OA) of up to 93% at pixel-level using polygon mask prompts, while using point and box prompts leads to an OA of 81% and a mF1-score of 72%. Experiments with other Sentinel-2 composite images do not show significantly different results compared to R-G-B images. Incorporating data from different time steps for map updating shows good results, but remains inconclusive due to the small amount of change in the dataset.