Using GANs and Facial Landmark Detection for Virtual Reality Conferencing

This work aims to explore an alternative approach to traditional virtual reality-based video conferencing methods, such as 3D heads and avatars. Instead of these complex representations, our method animates a single image of a conference participant. By live-recording the participant's mouth and face, even when they are wearing a headset, we enable mouth movements. This animation process is achieved using a combination of vid2vid, GANs, and motion transfer. The prototype was evaluated by 10 participants, who assessed the sense of presence they experienced within the virtual environment. The results indicate a positive response, with participants reporting a strong feeling of presence and engagement. These findings suggest that this prototype has promising potential as a foundation for future development.