Domain generalized remote sensing scene captioning via country-level geographic information

In this study, we explored the performance impact of incorporating country-level text-based geographical information into a large-scale vision language model, fine-tuned for the captioning of optical remote sensing images. We hypothesized that a model trained with country-level textual geographical context along with visual scenes would enhance its captioning capabilities when confronted with images from previously unseen countries or even continents, coupled with their respective geographical context. A large language and vision assistant (LLaVA) was fine-tuned using optical images from European countries and tested on images from other continents to evaluate its generalization capabilities. Here we report results of experiments conducted across 175 countries via the newly published Skyscript dataset, demonstrating that even superficial geographical information obtained from Wikipedia articles can mitigate the cross-country domain shift by several points in terms of accuracy score. This multimodal approach, combining textual geographical context with visual data, shows significant potential for improving the generalization capabilities of vision language models in tasks involving diverse and previously unseen geographical regions.