Lighting Estimation Guided Portrait Relighting Based on Diffusion Model

The relighting task requires decoupling the lighting from the scene, which can relight people or objects in the scene for applications such as mixed reality. In this paper, we design a novel diffusion-based portrait relighting method guided by lighting estimation. In the process of portrait generation, in addition to the portrait in the original scene and the background image of the target scene, the light map estimated from the background image of the target scene is also input and used as the control condition related to lighting, which guides the diffusion model to generate the relighted portrait that the material is consistent with the input portrait and the lighting is consistent with the target scene. Compared to previous portrait relighting methods, this method explicitly and more accurately estimates and decouples lighting using a joint lighting expression model to generate relighting portraits that are more accurately lit and more realistic. In addition, in order to solve the problem that portrait relighting datasets are difficult to obtain, this paper constructs a high-quality portrait relighting dataset based on real portrait datasets and image generation techniques. The experimental results show that the proposed method is capable of generating relighting portraits with a high sense of realism and accurate lighting, in which the guidance provided by the lighting estimation substantially improves the relighting results.