Action Recognition System Using Full-body XR Devices for Sports Metaverse Games

In this paper, we propose a new action recognition system using full-body XR devices for sports metaverse games. The proposed system simultaneously utilizes a XR headset, XR controllers and XR trackers to obtain full-body motion data of a XR user and exploits these full-body motion data for playing sports metaverse games with the 3D avatar of the XR user. A recurrent neural network model with bidirectional gated recurrent units is trained to segment and classify full-body motion data into sports actions for XR users of different generations and genders. For evaluation, the proposed system is applied to golf in practice. The performance of the proposed system is verified with real experiments.