Fusing enhanced YOLO and knowledge graph-based large language models for automatic risk perception in tower crane operations

Tower crane operations in construction are inherently hazardous due to complex and dynamic site environments. Enhancing operators' perceptual and cognitive capabilities is essential for ensuring safety and improving situational awareness. This paper presents an integrated framework that combines an improved YOLOv8 model with a Knowledge Graph (KG)-enhanced large language model to achieve proactive and intelligent safety management. The improved YOLOv8 incorporates attention-based optimization to improve detection accuracy for small targets in tower crane perspectives. A domain-specific safety KG is constructed to represent critical entities, relationships, and operational contexts, and is aligned with a fine-tuned GPT model, enabling semantic reasoning and context-aware hazard interpretation. The integrated system links visual perception with structured knowledge reasoning to provide real-time and interpretable safety feedback. This approach enhances the perception, understanding, and decision-making capabilities of tower crane operators, transforming safety management from reactive monitoring to proactive and intelligent control in complex construction environments. • Integrated CV-KG LLM framework for tower crane safety. • YOLOv8 + CBAM improving small-object detection accuracy. • Domain-specific safety KG (100 nodes, 355 relations) supports contextual reasoning. • KG-aligned fine-tuned GPT enabling real-time and explainable risk interpretation. • F1 score of 89%, safety management transformed to proactive mode.