An Interactive Desktop Application for Human Gesture-Controlled Music Generation Using Deep Learning-Based Hand Tracking

This paper presents an interactive desktop application for human gesture-controlled music generation. The system uses a standard webcam and deep learning-based hand tracking to detect both hands and convert hand features into musical control actions. The application was developed in Python and uses OpenCV, MediaPipe, FluidSynth, MIDI synthesis, and WAV stem playback. It provides three modes. Free Play lets the user control notes and dynamics with hand movement. Song Generation lets the user guide melody, bass, and chords inside a beat-based structure. Stem Control lets the user activate or mute separate audio stems with finger gestures. Deep learning is used for hand tracking, while musical output is produced through rule-based mappings, MIDI synthesis, and prerecorded WAV stem playback. The system was tested with 30 voluntary participants. The average frame rate was 21.81 fps, and the average MediaPipe inference time was 20.80 ms. The average calibration score was 7.93 out of 8 checks. In Stem Control, an average of 4.80 out of five software-defined task criteria were met. The software-defined beat-drop timing condition was met for 16 of the 30 participants. Questionnaire scores were positive, with 4.57 for ease of use, 3.97 for naturalness, and 4.47 for reuse intention. The results show that webcam-based hand tracking can support interactive music control without requiring specialized hardware.