Beyond Hearing: Investigating Embodied Sound Experiences in Virtual Reality Music Videos

In virtual reality music video (VRMV) production, sound design transcends traditional roles, enhancing immersive and embodied experiences unique to the VR medium. Unlike conventional music videos, where sound primarily complements visuals, VR’s spatial potential introduces sound as a distinct creative element integral to user engagement. This research explores how VR’s immersive qualities open new avenues for sound design that practitioners can harness to amplify perceptual experiences, highlighting VR's unique affordances beyond merely translating 2D content to 3D spaces. Through a series of experimental studies, this work investigates specific sound design strategies to enhance immersion and embodiment in VRMVs. In the initial phase, focus group discussions and expert interviews revealed that immersive sound design could strengthen spatial awareness and interactivity and evoke realistic bodily sensations similar to live concert experiences. These insights informed the next phase, where experiments examined the spatial perception of sound within a VR environment, introducing the concept of "spatial attributes" and positing that the overall spatial perception of a scene is composed of the spatial attributes of each individual sound unit. Additionally, experiments identified spatial differences between various types of sound units in virtual spaces. Further studies explored the embodiment of sound, simulating bone conduction to evoke physical sensations of sound resonance, especially in concert-like settings. Results showed that such simulated sensations intensified sound realism, heightening user immersion. Subsequent experiments incorporated interactive design into this exploration and extended it to dynamic sound contexts, where moving sounds combined with visual and environmental cues increased interactivity while significantly enhancing physiological responses, as measured by Galvanic Skin Response (GSR), despite participants’ unawareness of sound shifts. Finally, a VRMV prototype was developed to suggest the real-world applicability of these sound design strategies, illustrating their potential for enhancing VRMV production. This study highlights the importance of sound as a medium-specific tool in VR, advancing both theoretical understanding and practical applications for VRMV practitioners. In summary, this research provides innovative perspectives on VRMV sound design, suggesting that VR’s immersive qualities should not merely replicate traditional media experiences but instead introduce novel creative dimensions that expand the audiences’ sensory engagement. The findings offer a foundation for future research and practical insights for VRMV creators aiming to fully leverage VR’s transformative potential.