NePO: Neural Point Octrees for Large‐Scale Novel View Synthesis

Abstract Point‐based radiance field rendering produces impressive results for novel‐view synthesis tasks. Established methods work with object‐centric datasets or room‐sized scenes, as computational resources and model capabilities are limited. To overcome this limitation, we introduce neural point octrees (NePOs) to radiance field rendering, which enables optimisation and rendering of large‐scale datasets at varying detail levels, including different acquisition modalities, such as camera drones and LiDAR vehicles. Our method organises input point clouds into an octree from the bottom up, enabling level of detail (LOD) selection during rendering. Appearance descriptors for each point are optimised using the RGB captures, enabling our system to self‐refine and address real‐world challenges such as capture coverage discrepancies and SLAM pose drift. The refinement is achieved by adaptively densifying octree nodes during training and optimising camera poses via gradient descent. Overall, our approach efficiently optimises scenes with thousands of images and renders scenes containing hundreds of millions of points in real time.