Real-Time Gesture Classification via Multi-Modal Sensor Data for Intuitive Performance Mapping

We present an approach to real-time continuous gesture recognition via accessible classification and regression tools and its application in an embodied musical performance workflow. By combining electromyogram (EMG) muscle signals with gyroscope and accelerometer data from a wireless inertial measurement unit (IMU) system, we attained well-rounded descriptors of ongoing gestural arm movements. By training on this data via multilayer perceptron classification and outputting confidence ratings in place of predicted classes, we successfully detected five pre-chosen gesture types in real time and interpolated smoothly between their associated audio clips. Integrating this model into an interactive performance system let us harness these confidence ratings as overlapping influences on a musical arrangement, expanding on embodied musical associations via intuitive aesthetic mappings. Our results demonstrate the feasibility of continuous gesture recognition with the current generation of accessible machine learning tools, extending prior research into new use cases while enabling exploration and application of embodied expressive associations.