HFVR: Free-Viewpoint Reconstruction of Moving Humans from Monocular Video

In AR/VR and other application scenarios, the quality of human free-view video reconstruction significantly impacts user experience. Current reconstruction methods often rely on images captured by multi-view cameras and struggle to handle deformations caused by human movement. To address this issue, we propose HFVR (Human Free-Viewpoint Reconstruction), which aims to reconstruct the free view of moving humans from monocular videos. The core concept of HFVR is to utilize the motion deformation field to connect the observation space and the canonical space to perform a 3D representation of humans in the canonical space for rendering from any perspective. We first introduce the motion deformation field to transform the human body in the observation space into a standard space, then model the human body in that standard space. Second, we design a 3D global perception module to use global structural information to address deformation accuracy issues. Finally, we construct a rendering-guided module based on a reference frame that effectively overcomes limitations due to a lack of multi-view information by combining 2D observation. Extensive experiments demonstrate that our method can render realistic free-viewpoint motion humans using monocular videos. Our method improves the LPIPS (Learned Perceptual Image Patch Similarity) metric by 9.89% compared to that of HumanNeRF, as well as by 45.28% compared to that of Neural Body.