Automated knowledge elicitation for rail vehicle identification using vision-language models

• Automated framework elicits rail-specific knowledge into system prompts. • Few-shot knowledge elicitation adapts VLMs for rail vehicle identification. • Error-driven visual–textual attributes enhance rail-domain class discrimination. Rail transportation accounts for a significant portion of long-distance freight movement. Existing datasets typically rely on aggregated reports. Recent studies have explored vision-based deep learning models to capture details of locomotives and railcars. However, training an accurate supervised deep learning model requires a labor-intensive data annotation. Vision-language models (VLMs) have shown strong potential in general-purpose visual recognition tasks by aligning image regions with textual descriptions. However, their performance in specialized transportation domains often declines without domain-specific knowledge refinement. Current methods typically rely on labor-intensive crafted prompts or manual incorporation of expert knowledge. To address this limitation, this study proposes Knowledge Auto-Elicitation for System Prompts (KAESP), a novel framework that systematically extracts domain knowledge from identification errors and iteratively refines system prompts, which enables adaptation without model retraining or parameter updates. KAESP integrates automatically elicited textual knowledge with few-shot visual examples to create a system prompt that bridges visual grounding and domain expertise to enhance fine-grained railway vehicle identification. Experimental evaluations were conducted on a real-world dataset with 24 locomotive and railcar classes collected in Barstow, California. The results demonstrate that KAESP improved the average F1 score from 0.77 with only few-shot in-context learning to 0.82, with 14 classes achieving F1 scores above 0.80. Significant improvements were observed for several railcar categories, such as Container on Flatcar Double-Stacked with one 53-foot and two Twenty-foot Equivalent Unit containers F1 score improved from 0.41 to 0.61, and Gondola increased from 0.66 to 0.80. The results show that auto-elicitation knowledge significantly enhances VLM performance on rail train understanding.