Capabilities of Gemini, GPT, and DeepSeek for transportation in Dutch: Assessing domain-specific and underrepresented language challenges

Large Language Models (LLMs) have showcased remarkable progress in language comprehension and interactive decision-making, primarily in English and across general tasks. However, their capabilities in domain-specific reasoning and underrepresented languages remain underexplored. This study investigates the performance of LLMs in the Dutch language within the transportation domain, with a focus on cross-lingual generation and multimodal input processing. It aims to understand LLM's model behavior in the transportation domain to help select appropriate foundation models for further fine-tuning on specific tasks when training from scratch is not feasible due to data availability or high costs. We evaluate the performance of nine different LLMs from three prominent open- and closed-source LLM families: Gemini, GPT, and DeepSeek, using 972 questions distributed across six datasets. These datasets vary in complexity, input modality, and in English or Dutch to comprehensively assess the models. Our findings show that while LLMs generally perform better in English, several models demonstrate strong cross-lingual capabilities in Dutch. The DeepSeek R1 and GPT-4o models perform relatively well in both languages, while Gemini 1.5-Pro struggles with Dutch content. Additionally, LLMs exhibit a solid understanding of the context of transportation and regulation by effectively interpreting both information from images and text to answer domain-specific questions. However, they still lag behind the performance of elementary children in comprehensive transportation-related tasks, suggesting that LLM reasoning under zero-shot conditions may be limited. This study underscores the potential of LLMs in domain-specific applications and underrepresented languages and supports informed selection of foundation models for fine-tuning in these context settings.