Feature Field Fusion for few-shot novel view synthesis

Reconstructing neural radiance fields from limited or sparse views has given very promising potential for this field of research. Previous methods usually constrain the reconstruction process with additional priors, e.g. semantic-based or patch-based regularization. Nevertheless, such regularization is given to the synthesis of unseen views, which may not effectively assist the field of learning, in particular when the training views are sparse. Instead, we propose a feature Field Fusion (FFusion) NeRF in this paper that can learn structure and more details from features extracted from pre-trained neural networks for the sparse training views, and use as extra guide for the training of the RGB field. With such extra feature guides, FFusion predicts more accurate color and density when synthesizing novel views. Experimental results have shown that FFusion can effectively improve the quality of the synthesized novel views with only limited or sparse inputs. • A novel NeRF method using spatial feature maps to improve sparse view synthesis. • Concurrently learning both feature & color fields with pixel-level loss supervisions. • A fusion scheme regularizing color prediction with learned features as condition guide. • Extensive tests validate the performance and model-agnostic advantages of the method.