Simulation vs. Hallucination: Assessing Vision-Language Model Question Answering Capabilities in Engineering Simulations

Engineering simulations generate complex multimodal data that are crucial for design iteration and validation. The interpretation of these simulations traditionally requires significant domain expertise and cognitive effort. Recently, vision-language models (VLMs) have demonstrated impressive capabilities in general-domain multimodal reasoning tasks, offering the potential for automating simulation data interpretation. However, the effectiveness of these models in specialized engineering contexts remains largely unexplored. This paper presents an initial comparative evaluation of state-of-the-art VLMs on question answering tasks involving structural and fluid dynamics simulation data across three modalities: text, images, and videos. In doing so, we introduce a domain-specific benchmark dataset comprising true/false questions testing comprehension for engineering simulations. Unlike general-purpose multimodal benchmarks, our evaluation focuses specifically on the technical interpretation of engineering simulation outputs, requiring specialized domain knowledge and physical reasoning that is absent from broader multimedia assessments. Our results demonstrate that text modality yields substantially higher performance (up to 69.2% accuracy with GPT-4o, 66.3% with LLaVA) than visual inputs (52.9-55.8%), with GPT-4o, LLaVA, and Phi-3 exhibiting the strongest capabilities for text comprehension. Models on average performed better on structural analysis tasks than fluid dynamics problems, with minimal advantage observed for native video processing over batched image approaches. However, reliability is still far below that needed for engineering applications, highlighting significant challenges in applying current VLMs to the interpretation of engineering simulations.