Redefining Hybrid Meetings through Immersive Digital Twins with Avatar Driven Physical Representation and Dynamic Remote Interaction

As the landscape of modern communication continues to evolve, there is a growing need for cutting-edge virtual video conferencing technologies. In response, our study proposes a novel method for developing a system that replicates a hybrid meeting environment as a digital twin which remains a relatively unexplored yet promising direction in immersive communication research. Remote participants access this virtual meeting setup through a web platform composed of two separate windows, one displaying their own live video and the other presenting the virtual meeting space. Remote users accessing the system through a web browser are given the autonomy to explore the virtual meeting space from any viewpoint they prefer. Unlike current platforms, our solution also allows remote users to choose their appearance location on monitors within the virtual space and this feature along with spatial awareness is missing in the current systems. The primary purpose of this research is to address the lack of immersion, spatial interactivity and realism in current video conferencing systems. Furthermore, this work demonstrates the feasibility of creating an immersive digital twin of a hybrid meeting setup, highlighting its potential to enhance user presence and interaction in remote collaboration environments. While widely adopted platforms offer reliable communication, they often limit participants to static viewpoints and provide minimal representation of physical presence. Additionally, we demonstrate the feasibility of using a neural network-based rigging pipeline to animate avatars that represent physical participants in real time. Python and MediaPipe were utilized for real-time body tracking, enabling seamless avatar control. Since we are implementing everything from scratch, we have full control over every customizable feature and functionality with the Agora video SDK based on WebRTC. Thorough network performance analysis was conducted to ensure the reliability and responsiveness of the system. Contributions highlighted in this paper extend the concept of hybrid meetings by bridging the gap between remote presence and physical representation within virtual spaces, ultimately advancing the field toward more human-centric and immersive conferencing technologies.