KOMPSAT-3/3A Image-text Dataset for Training Large Multimodal Models

This study aims to improve the accuracy and interpretability of large multimodal models (LMMs) specialized in satellite image analysis by constructing an image-text dataset based on KOMPSAT-3/3A imagery and presenting the results of training using this dataset. Conventional LMMs are primarily trained on general images, limiting their ability to effectively interpret the specific characteristics of satellite imagery, such as spectral bands, spatial resolution, and viewing angles. To address this limitation, we developed an image-text dataset, divided into pretraining and finetuning stages, based on the existing KOMPSAT object detection dataset. The pretraining dataset consists of captions summarizing the overall theme and key information of each image. The fine-tuning dataset integrates metadata -including acquisition time, sensor type, and coordinates- with detailed object detection labels to generate six types of question-answer pairs: detailed descriptions, conversations with varying answer lengths, bounding box identification, multiple choice questions, and complex reasoning. This structured dataset enables the model to learn not only the general context of satellite images but also fine-grained details such as object quantity, location, and geographic attributes. Training with the new KOMPSAT-based dataset significantly improved the model’s accuracy in recognizing regional information and object characteristics in satellite imagery. Finetuned models achieved substantially higher accuracy than previous models, surpassing even the GPT-4o model and demonstrating the effectiveness of a domain-specific dataset. The findings of this study are expected to contribute to various remote sensing applications, including automated satellite image analysis, change detection, and object detection.