Remote Sensing Semantic Segmentation Quality Assessment Based on Vision Language Model

Variations in scene complexity and image quality across remote sensing images lead to inconsistent performance when applying pre-trained semantic segmentation models. To ensure quality control for downstream tasks, quality assessment on interpretation results becomes a challenging task, primarily due to the limited availability of reference annotations. To address this issue, we propose RS-SQA, a remote sensing semantic segmentation quality assessment model based on the vision language model (VLM) for images without ground truth. RS-SQA uses a dual-branch architecture that leverages a RS VLM for semantic understanding and utilizes intermediate features from segmentation methods to extract implicit information. We fuse these two feature streams via a semantic-guided module, yielding accurate quality predictions. Specifically, we introduce CLIP-RS, a large-scale pre-trained VLM trained on 10M remote sensing images with purified captions. Feature visualizations confirm that CLIP-RS can effectively differentiate between various levels of segmentation quality. To facilitate the development of RS semantic segmentation quality assessment, we present RS-SQED, a new dedicated dataset sampled from 4 major RS semantic segmentation datasets and annotated with segmentation accuracy scores from 8 representative segmentation methods. Experiments on the established dataset demonstrate that RS-SQA outperforms state-of-the-art quality assessment models and aids in selecting the most suitable pre-trained models for remote sensing semantic segmentation by comparing predicted accuracies across different models.