Exploiting 3D Semantic Priors for Ultra-Low Bitrate Human Video Compression

The recent advancements in model-based video codecs have revealed significant potential. However, encoding human videos at ultra-low bitrates remains an unprecedented challenge due to the complex structure and dynamics of the human body. Inspired by the recent 3D semantic prior-based face video compression, we make the first attempt to exploit 3D prior information for extreme human video compression. More specifically, at the encoder, we first extract 3D human model parameters with specific semantics from each frame and then compress the keyframe and the parameter residuals of adjacent frames. Finally, utilizing the improved motion imitator, the motion information is derived from the decoded keyframe and reconstructed 3D prior representations to synthesize each frame. Experimental results indicate that the proposed scheme can produce satisfying reconstruction results at ultra-low bitrates and better support downstream 3D tasks such as constructing 3D human models at the same bitrates.