ImproVision: Visual Communication and Human-Computer Interactions for Musical Creativity

Abstract This paper presents the ImproVision framework for ubiquitous creative human-machine interaction through nonverbal musical communication. The authors explore visual co-creativity through two case studies: one in which a pan, tilt, zoom (PTZ) camera interprets musician gestures to guide harmonic changes, acting as a robotic conductor; and another wherein gestures modulate parameters of Somax2, an AI improvisation agent. The authors underscore flexible, modular design and the importance of intuitive visual communication channels. Prospective future directions include refining the gesture toolkit and collecting data to better understand human-machine co-creativity. The authors’ findings contribute to a broader understanding of machine intelligence in augmenting human creativity, particularly in musical settings.