Investigating Activity-Linked Audio for Conveying Partner Activity Information in Collaborative VR

Abstract In collaborative virtual reality (VR), users need information about a partner's presence, location, and activity, but visual access is limited to the head-mounted display's field of view and sounds arising from bodily activity are not necessarily reproduced. We investigated activity-linked audio, defined as non-speech auditory cues linked to avatar activity, including sounds triggered by specific activities and sonification varying with activity state. We compared three conditions: Silent, with no activity-linked audio; Footstep, which presented footstep sounds during walking; and Noise, which mapped an avatar's overall activity level to red-noise volume. Twenty-four participants in 12 pairs performed a collaborative block-assembly task while alternating between Director and Executor roles. We evaluated social presence, affective evaluation of the partner avatar, task performance, head orientation, body movement, and interpersonal distance. Behavior was analyzed across five task phases and by whether the target block was Front, Back, or Lateral relative to the Director. No consistent condition differences were found in condition-level subjective evaluations, task performance, or head orientation toward the partner. Executor body movement showed Condition × Phase × Direction interactions. During the search-and-selection phase, when the Executor worked behind the Director, the proportion of time moving was higher in Footstep and Noise than in Silent, while movement distance per trial was greater in Footstep than in both Silent and Noise. Thus, differences among auditory conditions depended on task phase and spatial relationship, particularly during search behind the Director.