Human Assembly Action Recognition Framework for Real-World Assembly Operation

Abstract In recent years, significant progress has been proceeded in human action recognition (HAR) technology. When applying HAR to practical scenarios, especially in the task of recognizing Standard Operating Procedures (SOP) for factory operators, some challenges such as resource requirements, differentiation of diverse movements, and difficulties in manual annotation exist. In this research, an assembly action recognition framework (AARF) was proposed to recognize the assembly operation conducted at the station level using the SlowFast video understanding model. First, the pre-trained YOLOv7 model is employed as a non-assembly action filter to detect the main operator, and then the irrelevant background is masked for a refined region of interest. Then, the preprocessed data is fed into SlowFast for action recognition through both temporal and spatial features. Finally, the confidence calibration strategy is implemented to enhance the robustness of results. A series of ablative experiments were conducted to study for optimizing the selection of model parameters, incorporating attention mechanism, non-assembly action filtering, and recognition result filter. Multiple assembly motion videos were collected from two similar product lines at a real-world production line. According to experimental results, the proposed method can improve the accuracy of human motion recognition on real factory production lines by up to 86.3%.