Open-Vocabulary Segmentation for Remote Sensing

Open-vocabulary segmentation uses natural language class descriptions for zero-shot segmentation. In remote sensing, datasets are limited since manual segmentation requires expertise. The success of vision-language models (VLMs) in openvocabulary classification and retrieval suggests using ground images as intermediates can bypass this limitation. VLMs trained with contrastive loss show strong correspondence between vision transformer (ViT) patches and language embeddings. We leverage this with SegViT architecture to generate segmentation masks. Specifically, we use a ViT backbone trained on a patch-wise contrastive learning objective to match CLIP model image embeddings to the patch location. We replace the learned class embeddings in the SegViT decoder with language embeddings of class descriptions, and fine-tune the model on the OSMSeg dataset. We also evaluate GRAFT, SegViT, and CLIPSeg on OSMSeg to present baselines for this task and suggest improvements for future advancements.