Image‐Based Spatio‐Temporal Interpolation for Split Rendering

Abstract Low‐powered devices – such as small form factor head‐mounted displays (HMDs) – struggle to deliver a smooth and high‐quality viewing experience, due to their limited power and rendering capabilities. Cloud rendering attempts to solve the quality issue, but leads to prohibitive latency and bandwidth requirements, hindering use with HMDs over mobile connections or even over Wifi. One solution – split rendering – where frames are partially rendered on the client device, often either requires geometry and rendering hardware, or struggles to generate frames faithfully under viewpoint changes and object motion. Our method enables spatio‐temporal interpolation via bidirectional reprojection to efficiently generate intermediate frames in a split rendering setting, while limiting the communication cost and relying purely on image‐based rendering. Furthermore, our method is robust to modest connectivity issues and handles effects such as dynamic smooth shadows.