Fast Few-Shot Key Gesture Spotting Tool with Fewer Sensors and Frames

Accurate and timely identification of key gestures is fundamental to many virtual reality and metaverse applications. Our goal is to address the demand for expertise in areas such as deep learning that developers are confronted with when integrating key gesture spotting into their VR applications. Previous methods relied on multiple sensors or camera arrays to collect gesture signals, which lead to uncomfortable user experience. However, simply removing sensors often results in performance collapse. In this paper, we propose a novel few-shot framework for key gesture spotting that can be integrated into VR applications, allowing developers and users to create their own key gestures as triggers to perform specific behaviours. Our method involves using high-resolution gesture sequences as sample inputs during training, encoding and storing them as latent representations. During evaluation, low-resolution sequences are input and matched for similarity with the representations in the memory bank. We then compute weights and perform a weighted sum of both, aiming to provide additional information for the low-resolution inputs. Experiments show that enhancing the cosine similarity classifier with meta-learning and a memory bank achieves near state-of-the-art accuracy efficiently, with minimal overhead, and can be achieved using only a head-mounted VR device.