Hand-in-river: Design of direction-dependent virtual cues of assistive/resistive force via auditory displacement and pitch variation

This paper investigates the role of sound-only manipulations in eliciting pseudo-force perception during lateral hand movements within a virtual river environment. Building on prior formative and exploratory studies, a pseudo-force feedback design is refined, leveraging two mechanisms: (i) auditory displacement, a spatial shift in the virtual, binaural position of the hand-in-river sound, creating a conflict between where the hand is felt and where it is heard; and (ii) pitch variation, making the sound splashier or calmer by raising or lowering its pitch, relying on learned touch–sound associations. No physical force is actuated and no haptic or kinesthetic response is measured; the manipulation is entirely auditory and the dependent measure is a forced-choice judgment. A within-subjects psychophysical experiment used a two-interval forced-choice (2IFC) protocol with a 2 × 2 factorial design, evaluating pitch variation and displacement for resistance (leftward movement against the flow) and assistance (rightward movement with the flow). Results showed that both restricting displacement and pitch increase reliably biased participants toward perceiving resistance, with pitch variation being the dominant factor—suggesting that spectral content carries more perceptual weight than localization—and no interaction between cues, indicating an additive effect. Conversely, aiding displacement and pitch decrease did not reliably bias participants toward perceiving assistance—though displacement exhibited a positive trend—revealing a potential direction-dependent asymmetry. These findings refine the design space of sound-based pseudo-force feedback for inclusive immersion among visually impaired users. Resistive cues can be tuned modularly to create pseudo-haptic boundaries for navigation, while assistance likely requires strategies beyond simply inverting resistive cues.