Real‐Time Mapping of High‐Degree‐of‐Freedom and High‐Fidelity Gestures Based on Piezoelectric Sensor Clusters

ABSTRACT As humanity advances into the virtual world and the era of mixed reality, high‐fidelity gesture mapping has become a key interface connecting the physical world and the digital space. However, the existing solutions are limited by the bottlenecks of active sensors in terms of information richness and structure and have insufficient precise reconstruction capabilities for high‐degree‐of‐freedom hand movements, making it difficult to achieve a natural and seamless interaction experience. Here, a dynamic gesture mapping system (DGMS) has been developed, which combines a lightweight and efficient hand motion signal acquisition device (HSAD) with an efficient signal processing algorithm (SPA). HSAD captures the motion information on 17° of freedom for the hand through sensor clusters and wirelessly transmits the data to the signal processing platform. On this platform, SPA converts electrical signals into bending angle information, with an error below 1°. DGMS synchronously and accurately maps the hand movement information output by SPA to the virtual environment, achieving real‐time interaction with objects in the virtual space. This study has for the first time achieved such a high‐degree‐of‐freedom and high‐fidelity gesture mapping using passive sensors and explored the application of gesture mapping in the virtual–real coexisting intelligent enhancement platform, offering a novel technical route for digital twin applications.