Proposal for a Gesture Manipulation System with Drumsticks Using Video Recognition Technology

In this study, we propose a system that enables gesture operation of an electronic musical instrument while holding drumsticks by utilizing video recognition technology. Electronic musical instruments allow manipulation of parameters such as pitch and audio effects. Conventionally, in order to operate an electronic musical instrument, it is necessary to release the drumstick from the hand and operate it directly, making it difficult to operate an electronic musical instrument during a performance. This system can recognize gestures and manipulate parameters of electronic instruments while playing. The challenges in this research are to develop gestures that are easy to learn and to reduce the impact of misrecognition on the performance. We developed a gesture recognition technique that uses features created from the detection of drumstick and hand landmarks. We also developed recognizable gestures based on stick trick and drumstick waving motions. In an evaluation experiment with three drummers as subjects, five of the six gestures were found to be easy to learn. Recognition accuracy was also evaluated using video images of the gestures being performed during the performance, and three of the gestures that were easy to learn achieved the target accuracy.