A Standardized Benchmarking Framework for the Automated Evaluation of AI-Generated Travel Plans

The rapid adoption of large language models (LLMs) transformed the way users obtained travel recommendations and generated personalized travel plans. However, standardized methods for comparing AI-generated travel itineraries across heterogeneous providers remained limited. This study presented the design, implementation, and experimental evaluation of a web-based benchmarking framework for the standardized and automated comparison of AI-generated travel plans. The framework integrated standardized prompt generation, structured JSON response processing, field-level comparison, deterministic rule-based scoring, model ranking, and exportable benchmarking reports. It was evaluated across three travel-planning scenarios involving Prague, Rome, and Tokyo, with three independent benchmark executions attempted for each scenario, resulting in 27 provider-level requests. Eight executions produced complete three-provider comparisons; one Rome execution was incomplete because Microsoft Phi returned malformed JSON. Across the eight complete executions, Gemini achieved a mean structural-completeness score of 57.00/57 (SD = 0.00), ChatGPT achieved 47.38/57 (SD = 3.02), and Microsoft Phi achieved 34.63/57 (SD = 5.26), with the same rank ordering observed in all complete comparisons. Supplementary constraint-compliance analysis demonstrated that structural completeness did not necessarily imply full satisfaction of travel requirements. Sustainability-related outputs were additionally examined as a separate dimension. The results demonstrated consistent framework operation under the tested scenarios and repetitions while emphasizing the need to distinguish structural completeness from constraint compliance, factual accuracy, practical feasibility, and sustainability-related performance.