Real-Time Motion Augmentation and Synthesis for Animating the Hands and Eyes of Virtual Humans and Avatars

Virtual Reality (VR) enables users to interact within virtual worlds via an embodied virtual representation of themselves called an “avatar”. Because avatars are essential for immersive experiences, it is important to consider how altering or augmenting avatar motion affects virtual experiences. This dissertation aims to improve virtual experiences by addressing some of the many challenges in animating avatars and virtual humans. In our first study, we addressed the lack of tactile feedback during virtual grasping by using visual feedback techniques. We augmented the avatar’s hand motion to remain outside virtual objects (“outer hand”) even when the user’s hand penetrated them. Our study revealed that while this reduced grasping performance, users preferred it over allowing hand penetration. Our second study incorporated auditory feedback alongside visual feedback to examine its impact on interaction performance and ownership. Despite a decrease in performance, users preferred the addition of audio feedback. Gaze behavior plays an important role in avatar-mediated communication. Therefore, we developed a model to synthesize eye motion during two-party conversations in VR using audio and head movement data. We conducted a user study with results that provide insight into expected conversational gaze behavior, as well as the complexities of animating untracked motion. Finally, we explored motion synthesis for applications requiring precise hand movements, such as guitar playing, a skill requiring precision in both movement and timing. We developed a model that synthesizes fretting hand animation for guitar using annotated music. Applications include virtual concerts, music instruction, and other dexterous tasks beyond music.