Evaluating Human Intuition, Multimodal AI, and an XGBoost Model for Estimating Ambient Population Density: A Questionnaire-Based Benchmark of Accuracy, Reasoning, and Confidence

The physical environment of a neighborhood, such as its road networks and land-use distribution, appears to be closely related to its ambient population density, but this relationship is difficult to measure quantitatively. A previous study based on Japanese neighborhoods addressed this challenge by developing an XGBoost baseline model to estimate Average Hourly Ambient Population Density (AHAPD) at a neighborhood scale using 16 physical environment-related features, achieving 75.9% accuracy. This study evaluates that baseline model by comparing it with human evaluators and multimodal AI models in a spatial reasoning task based on AHAPD estimation. A questionnaire-based benchmark with 29 sets of questions was developed to compare the three evaluator types across accuracy, reasoning, and confidence. The online questionnaire was distributed from 8 June–22 September 2025. In total, 100 responses were analyzed, comprising 94 human evaluators, 5 free-tier versions of multimodal AI models, and 1 output from the baseline model. The baseline model scored 16 points, compared with a mean of 15.1 points for the human evaluators and 14.2 points for the tested AI models. Its performance was also comparable to human evaluators with domain expertise and contextual familiarity, who averaged 16.0 points. The comparison further revealed differences in feature importance, confidence patterns, and characteristic errors among the evaluator types. Selected discrepancy cases highlighted possible strengths and weaknesses of each approach. The main contribution of this study is a questionnaire-based diagnostic benchmark for comparing evaluator types with different characteristics in spatial reasoning tasks.