Conditional plane-based multi-scene representation for novel view synthesis

The method overview. The explicit representation on the right side represents the shared canonical space and the view space using 12 feature planes. The gray arrows indicate the feature projection from the canonical representation. The left side shows the deformation between the canonical space and the view space. Pairwise features (e.g., X Y − Z T ) from the canonical and view representations are aggregated to obtain the final feature vector. ⨂ indicates feature aggregation. Density and appearance decoders, which estimate the geometry and color, are conditioned on the scene’s latent s i . Existing explicit and implicit-explicit hybrid neural representations for novel view synthesis are scene-specific. In other words, they represent only a single scene and require retraining for every novel scene. Implicit scene-agnostic methods rely on large multilayer perception (MLP) networks conditioned on learned features. They are computationally expensive during training and rendering times. In contrast, we propose a novel plane-based representation that learns to represent multiple static and dynamic scenes during training and renders per-scene novel views during inference. The method consists of a deformation network, explicit feature planes, and a conditional decoder. Explicit feature planes are used to represent a time-stamped view space volume and a shared canonical volume across multiple scenes. The deformation network learns the deformations across shared canonical object space and time-stamped view space. The conditional decoder estimates the color and density of each scene constrained by a scene-specific latent code. We evaluated and compared the performance of the proposed representation on static (NeRF) and dynamic (Plenoptic videos) datasets. The results show that explicit planes combined with tiny MLPs can efficiently train multiple scenes simultaneously. The project page: https://anonpubcv.github.io/cplanes/ . • We present a novel multi-scene representation that uses twelve explicit feature planes. • The method uses encoder-less generalization to learn discriminative features for each scene. • We represent multiple dynamic scenes without relying on optical flow estimation. • The auto-decoded latent (the scene dimension) can interpolate between scenes. • It achieves state-of-the-art rendering results for both static and dynamic scenes.