NeFF: Semantic Information Learning In 3D Space Using Neural Feature Field

Inferring a 3D space with partial observation requires prior knowledge from previous experience, and deep learning offers an intuitive solution. Prior knowledge can be categorized into two extremes Global and Local. Global prior uses input image as a clue to retrieve seen intact 3D objects yielding 3D consistent inference with fewer details. In contrast, Local prior takes advantage of closeby views' overlapping areas and propagates input image cues to their vicinity via geometry transformation directly, producing realistic inferences around the reference views yet inconsistent inferences away from the reference views. We propose Neural feature field, an implicit 3D neural representation that combines the strength of the Global and Local prior. We represent each 3D scene with a latent embedding, then we perform a global scene-level prior to local view-specific feature prior with the neural feature field and volume rendering technique. Our model learns general scene properties across a category. Given a single input image at test time, the scene latent code is acquired via a two-round optimization process. Experiments on the SRN benchmark show that NeFF generalizes well to unseen objects within the same object category and is capable of producing 3D consistent inferences with sharp details from a single observation.