Hierarchical fine-grained action recognition for manufacturing applications

Recent years have seen a great development in human action recognition from video in various practical applications including autonomous work verification in manufacturing field, assembly lines specifically. One cycle of work usually includes a fixed number of consecutive instructed fine-grained actions (e.g. assembling steps) that need to be correctly recognized to ensure product quality. Previous works mostly employ a flat classification fashion which treats every action equally during training as well as inference regardless of their granularities. This poses great difficulties in case where the total number of actions is large and there are only limited visual cues to distinguish one action from another (i.e. [attach table leg] against [spinning table leg] amongst many other actions with respect to furniture assembling task). To remedy such problems, we propose a Hierarchical Action Classification scheme which groups to-be-classified actions according to their granularities and performs group-wise classification in a coarse-to-fine grained hierarchical way. Verification using the Desktop Assemble Dataset proved the effectiveness of our method in improving action classification performance in manufacturing context as compared to the flat classification counterpart.