TOWARDS BUILDING IMMERSIVE TELEPRESENCE EXPERIENCE

Recently, we have seen significant potentials of AR/VR technologies in transforming theway people interact with the world. This dissertation is focused on one of the most promisingapplications, telepresence, which allows users to feel like they are present at a location different from their physical location and is widely used in areas including medical therapy, remoteconferencing, and video games, etc.Reconstructing humans and objects in the virtual environment are two essential tasks in building an immersive telepresence experience. First, for reconstructing humans, we need to capturethe user’s body and estimate the human pose. Current solutions require either external camerasfrom third-person viewpoints or cumbersome wearable capture devices. In contrast, we presentEgoGlass, a pair of glasses equipped with cameras, as our AR/VR device, which is portable andadds next to no extra burden to the user. From images captured by EgoGlass, we propose a novelalgorithm for egocentric human pose estimation, overcoming issues of self-occlusion and incomplete body coverage caused by the cameras’ proximity to the human body. Second, for 3Dreconstruction algorithms running on AR/VR devices with constrained computational resources,computational efficiency is a major concern. To improve the efficiency of 3D reconstruction ofthe virtual environment without sacrificing accuracy, we introduce MVPSNet, which utilizes bothcamera viewpoint variations and lighting direction variations for high-fidelity 3D object recon-struction in a generalizable way. Our algorithm runs more than 400× faster than state-of-the-artapproaches with only negligible accuracy degradation. Third, we propose NBVNet for Next BestView selection in 3D reconstruction, improving 3D reconstruction efficiency by selecting the most informative viewpoint among a set of candidates. Our method outperforms existing methods with respect to the reconstruction quality, without requiring any prior knowledge, per-scene optimization, or pre-captured RGB / depth images.