Visual Instruction Tuning for Holistic and Regional Remote Sensing Imagery Comprehension

Recently, the visual instruction multimodal large language models (MLLMs) have been extensively studied in the nature scenario. However, current remote sensing (RS) MLLMs mainly focus on image-level understanding and typically allow interaction only through text instructions, resulting in limited accuracy and efficiency. To address those limitations, a visual instruction model is proposed in this article to extend MLLM’s fine-grained perception ability by incorporating bounding boxes with language instruction, aiming at achieving region-level visual understanding. To achieve this goal, a visual instruction dataset featuring multi-modal box-based region-text pairs is constructed. Furthermore, the visual instructions and images are encoded by the different encoders and subsequently fed into a large language model (LLM) along with text instruction tokens for tuning. Finally, experimental results demonstrate the proposed model’s promising performance in region-level image comprehension.