Fast and High-Resolution View Synthesis from a Single Input Panorama

We introduce a novel method to convert a single input panorama into a 3D colored mesh representation of the scene. Unlike recent methods based on neural rendering, which are limited to low-resolution inputs and offline rendering, our approach supports 4k resolution inputs and real time rendering in a virtual reality headset. We first estimate a depth map and produce an initial layered depth image (LDI) representation. We fill unseen regions behind objects by iteratively cutting and inpainting the LDI. We then convert the LDI into an optimized, texture mapped mesh to achieve a compact representation