Chain of Thoughts Based Prompting for Incident Detection Using Large Language Model

Warehouse safety is crucial but challenging due to the variety of incidents that occur, such as human errors and equipment failures. Traditional incident reporting often focuses only on the event itself, lacking context on the actions leading to the incident. To address this, this work introduces a system using video analysis to track and document the sequence of events before, during, and after an incident. This system integrates You Only Look Once (YOLO) models YOLOv9, YOLOv11 and Faster R-CNN (Region-based Convolutional Neural Network) to detect and annotate real-time events, capturing the entire incident sequence. Moreover, a language model automatically generates detailed and clear health and safety reports based on the detected actions, ensuring their accuracy and relevance. Testing demonstrated the system’s effectiveness in detecting key incidents like forklift mishandling and goods falling. YOLOv11 achieved a precision of 0.806 and a recall of 0.955, with a mean Average Precision (mAP) mAP50 score of 0.972. The system also showed strong sequence detection accuracy, with key events identified with a recall of 1.0 in some cases. Reports generated using Generative Pre-trained Transformer (GPT)-based models showed strong alignment with human-readable text, with a cosine similarity score of 0.874 and a Bidirectional Encoder Representations from Transformers (BERT) F1 score of 0.879. These results indicate that the system improves safety practices by providing comprehensive, actionable insights.