Equipment-centric workpiece localization in near real-time using deep learning-based vision and event-driven finite state machines

Continuous workpiece localization is critical for ensuring traceability, quality control, and process coordination in hot forging environments, where heavy workpieces are repeatedly transferred by specialized handling equipment such as overhead cranes among storage yards, furnaces, and forging stations. However, traditional workpiece-centric approaches are often unreliable under extreme temperatures, surface degradation, and irregular routing conditions, leaving a critical gap in continuous localization. To overcome these challenges, this study presents an equipment-centric framework that infers workpiece locations indirectly by analyzing equipment operations captured through video streams from multiple static 2D cameras. This paradigm shift from workpiece-centric to equipment-centric localization enables robust inference under harsh conditions by coupling equipment behavior with workpiece-handling events. The framework estimates floorplan-space 3D coordinates of handling equipment and recognizes workpiece-handling activities such as grasp and release. Event-driven finite state machines (FSMs) validate and detect discrete handling events based on equipment coordinates and recognized activity cues, enabling continuous inference and updating of workpiece states and floorplan-space coordinates. Experimental validation in an operational hot forging factory demonstrated the effectiveness and practicality of the proposed framework, achieving 100 % event detection accuracy within a 33-second tolerance window, a mean localization error of 317.8 mm, and a mean system latency of 21 seconds. Ablation studies further confirmed that integrating the Keypoint-Guided Attention (KPGA) mechanism improves activity recognition performance compared to baseline 3D convolutional neural networks (3D-CNNs) and transformer-based models. Beyond localization accuracy, the structured outputs of the framework bridge vision-based perception and event-driven reasoning by representing handling operations as interpretable state transitions. This capability enables data-driven visualization of workpiece transfers and quantitative evaluation of equipment handling states, contributing to more intelligent and traceable forging operations.