SEGUS: A Semantic Element Gesture Understanding System via Symbol-Path Decoupling

Gesture recognition plays a vital role in human-computer interaction. Most current gesture recognition systems infer gesture categories directly from raw inputs, often missing the semantic information embedded in the gesture execution process. This paper introduces a novel vision-based gesture semantic understanding system, referred as SEGUS. The goal of this system is to retrieve the symbol and path information contained in gestures. It utilizes a side-mounted camera on a wristband or smartwatch to capture video inputs and performs foreground-background separation of the video. The system extract is symbol and path information form the foreground and the background of T and video, respectively. Then, it achieves gesture recognition by comparing sequences of semantic elements. We evaluate the system through extensive experiments. The results indicate an average improvements from 6.68% to 16.39% in recognition accuracy compared to other end-to-end systems. Additionally, scalability tests show that by reorganizing existing semantic elements, the proposed system can accommodate newly added gestures without the need to re-collect sample data. This significantly reduces system overhead, achieving a 64.10% reduction in training time compared to methods requiring a training set for all gestures, with only a modest average accuracy decrease of 6.39%.