Correctness Comparison of Open-Weight and Proprietary LLMs for Spatial Tasks

Numerous benchmarks are used to rank Large Language Models performance in various tasks, as the popular open benchmarking platform Hugging Face Leaderboard shows. However, none of the widely cited benchmarks are specific to the unique challenges of the geospatial domain. A few benchmarks have been developed to fill this gap and applied to proprietary LLMs such as GPT, Gemini, and Claude. In this study, a standard set of tasks is run on three open-weight models (LLaMA-3, Mixtral, and Qwen-2), and their performance compared to pro-prietary models. We observe that the performance gap between the two types of models is not wide. This difference could be attributed to the orders of magnitude disparity in the number of parameters, with proprietary models typically having hundreds of billions of parameters, whilst the open models examined are in the 70–80 billion parameters range with the exception of Mixtral, which uses 39 billion parameters out of 141 billion (in Mixture of Expert models, some parameters are shared across experts).