Beyond Deterministic Mappings: Audiovisual Correspondence in Movement-Controlled Generative Music

Movement sonification transforms bodily motion into real-time sound for applications from motor rehabilitation to artistic performance. Designing effective movement-sound relationships requires balancing functional clarity with aesthetic engagement, a challenge that scales in complexity when using generative music as the sonification medium. This study explored how movement-sound correspondence with generative music models is affected by the interaction between movement characteristics and model-intrinsic variations. I compared perceived movement-sound congruence across three sound models (deterministic sawtooth, probabilistic drums, Markov-based generative piano) with systematic variations in arm movement amplitude (6.75°–94.5°) and hold time in a given position (0.25 sec–2 sec) using a 2D avatar. 26 participants rated audiovisual congruence for 36 stimuli. Deterministic mappings maintained consistent high congruence, while generative models showed strong movement-dependency: small amplitudes and long hold times elicited low congruence ratings, whereas larger amplitudes and moderate hold times were rated substantially higher. A significant crossover interaction (p<.001) revealed that movement amplitude and hold time affect perceived congruence in a complex and interdependent manner. This points to the importance of accounting for movement characteristics when working with sound models that introduce probabilistic or random variations to their outputs. These findings inform the design of movement-driven generative music systems by characterizing the perceptual constraints on audiovisual correspondence detection.