Theater of Future Puppetry: An Immersive Puppeteering Experience Based on Hands Tracking and Gestures Recognition

There are many studies on hand tracking and interaction with virtual puppets in the immersive experience.This research presents an advanced gesture recognition system called Theater of Future Puppetry.Unlike systems that only use hand tracking and hand skeleton manipulation of puppets, this system can recognize specific gestures used in traditional Budaixi (Taiwanese glove puppetry) and interact with them.The gestures were designed based on discussions with traditional Budaixi masters.Through this system, users can learn and understand the knowledge and cultural background of Budaixi puppet manipulation.