Tokenization matters for GEE code generation: a multidimensional tokenization benchmark

Large language models (LLMs) have been widely adopted to automate geospatial code generation in Google Earth Engine (GEE), lowering the barrier to conduct complex spatial analysis. While knowledge augmentation at the generation stage remains the dominant improvement strategy, inherent limitations in knowledge coverage and computational cost call for attention to more upstream components of the pipeline. Among upstream components, the tokenizer determines how inputs are segmented and represented before entering the model, yet its effect on code generation remains systematically unexamined. To fill this gap, we present GEEToken-Bench, the first benchmark dedicated to tokenization evaluation in the geospatial code generation domain. Built from 292,762 scripts, it provides 6,000 test instances across 12 functional categories. We assessed 21 tokenizers through 11 metrics spanning four dimensions—accuracy, sequence preservation, diversity, and efficiency. Results reveal a positive association between tokenization quality and code generation performance. WordPiece-based tokenizers excel at maintaining GEE API structural integrity, while byte-level BPE tokenizers favour compression efficiency over structural fidelity. These findings suggest that tokenization is positively associated with LLM-based GEE code generation performance, highlighting the tokenizer as a component that warrants greater attention in domain-specific design and optimisation in GeoAI.