Two-shot Shape and SVBRDF Reconstruction of Human Faces with Albedo-Conditioned Diffusion

Reconstructing 3D human heads with relightability has been a long-standing research problem. Most methods either require a complicated hardware setup for multiview capture or involve fitting a pre-learned morphable model, resulting in a loss of details. In our work, we present a two-step deep learning method that directly predicts the shape and SVBRDF of a subject's face given two images taken from each side of the face. We enhance SVBRDF prediction by first estimating the diffuse and specular albedo in image space, then generating texture maps in UV-space with a generative model. We also learn a 2D position map in UVspace for 3D geometry, eliminating the need for a morphable model. Contrary to single-image facial reconstruction methods, we obtain clear measurements on both sides of the face with two images. Our method outperforms state-of-the-art methods when rendering faces at extreme angles and provides texture maps that are directly usable in most rendering systems.