From hallucinations to hazards: benchmarking LLMs for hazard analysis in safety-critical systems

• Identifies gaps in current LLM benchmarks for hazard analysis. • Proposes a conceptual framework for creating specialized, domain-specific benchmarks. • Presents a pilot validation study with quantitative metrics on LLM performance. • Introduces Performance Consistency as a critical, new metric for evaluating LLMs in hazard analysis. • Issues a call for interdisciplinary collaboration on LLM benchmarks for safety critical systems. Integrating Large Language Models (LLMs) into safety–critical domains presents both promising opportunities and significant challenges for hazard analysis. While these AI systems demonstrate impressive capabilities in natural language understanding and generation, their reliability, accuracy, and trustworthiness in safety contexts remain critical concerns. This paper maps the current landscape of LLM benchmarks via a scoping review, categorizing them based on their primary focus and evaluating their applicability to safety–critical hazard analysis. The review reveals significant limitations in existing benchmarks, including inadequate coverage of safety-specific knowledge, limited evaluation of causal reasoning in technical contexts, the absence of regulatory compliance assessment, insufficient risk analysis capabilities, and minimal uncertainty handling evaluation. To address these gaps, a methodological blueprint for future benchmark development is proposed, tailored to hazard analysis in safety–critical systems. To test this blueprint, a pilot study was conducted across nine safety–critical scenarios, executed over three runs to measure performance consistency over time. This analysis of performance consistency across three evaluation runs provided initial evidence of significant volatility. While the model’s success rate in generating a response was stable, its analytical quality was inconsistent. Hazard identification scores varied between runs, and causal reasoning performance was consistently poor and unpredictable. This evidence of inconsistent analytical quality, even from a model of the same version, highlights a significant potential challenge for safety assurance.