Responsible AI in Construction Safety: Systematic Evaluation of Large Language Models and Prompt Engineering

Responsible artificial intelligence (AI) in construction safety requires rigorous, domain-specific evaluation. Deploying large language models (LLMs) without understanding their capabilities and limitations risks generating inaccurate information and compromising worker safety. This study benchmarked three leading LLMs—GPT-3.5, GPT-4o, and Gemini 2.0 Flash—using 385 Board of Certified Safety Professionals exam questions across seven knowledge areas. All models achieved passing scores (GPT-4o: 84.6%, Gemini: 80.8%, GPT-3.5: 73.8%), demonstrating strengths in the knowledge areas of safety management systems and hazard identification. However, LLMs performed relatively poorly in science and math, emergency response, and fire prevention. Error analysis identified four key limitations impacting performance: knowledge gaps, reasoning flaws, memory issues, and calculation errors. This study also highlights the impact of four prompt engineering factors, with variations in accuracy reaching 13.5% for GPT-3.5, 7.9% for GPT-4o, and 10.4% for Gemini. However, no single prompt configuration has proven universally effective across three models and seven knowledge areas, suggesting the need for model- and task-specific optimization. These findings establish critical benchmarks for the ethical and effective deployment of AI in construction safety, identifying areas where LLMs can support safety practices and where human oversight remains essential, and offering practical insights into optimizing LLM implementation through prompt engineering.