Training-Free Open-Vocabulary Segmentation of Aerial Drone Images Using Foundation Models

Semantic and instance segmentation have seen sig-nificant advancements, yet their application to aerial imagery faces numerous challenges. The high complexity of aerial images which often capture extensive scenes with numerous objects and varying object scales, differ considerably from the simpler natural images that the visual deep learning models are usually trained on. Additionally, the scarcity of annotated datasets for pixel-level semantic and instance segmentation in aerial imagery, hinders the development of specialized systems. However, re-cent progress in foundational models trained on vast, diverse datasets has improved generalization capabilities, making them robust across various scenarios. These models can be applied directly or adapted with minimal annotated data, significantly reducing the need for extensive labeled datasets. Even though foundation models are consecrated for their high generality, the results on specific scenarios, such as aerial images, are not as good as expected. In this paper, we provide an automatic panoptic segmentation and annotation system adapted to the characteristics of the drone vision images based on the visual foundation models Grounding DINO, SAM (Segment Anything Model) and the visual language model CLIP. We propose an algorithm that combines the foundation models and includes processing techniques to improve the performance on aerial scenes, eliminating the need for pixel-level annotated training data. The algorithm can segment objects using a fixed number of semantic classes or segment new ones through a text input by the user. We also provide a demo web application to test our solution on images from the UAVid dataset.