Portrait Video Compression with Semantic-guided Animation Model and Background Incremental Coding

The application of animation models in facial video compression has yielded significant coding gains, particularly at ultra-low bitrates. Despite notable advancements, research on portrait video scenes, especially in half-body and full-body contexts, remains underexplored. The mapping motion features and dynamic regions is often imprecise, leading to inaccurate motion parameters. Additionally, the reconstruction of occluded background is frequently suboptimal, as occluded regions lack ground-truth data. To this end, we propose a learning-based portrait video compression (LPVC) framework for half-body and full-body portrait videos with static backgrounds. Specifically, we first design a keypoint detector driven by human parsing to integrate semantic properties into the animation model. This facilitates richer features and enhances the motion features for dynamic areas. We further devise a background incremental coding scheme to reconstruct high-quality occluded backgrounds. The scheme incorporates background occlusion calculation and compensation modules to process the newly revealed background pixels between successive frames, eliminating redundant background transmission. Finally, a spatio-temporal portrait background fusion generative adversarial network (SPBF-GAN) is devised to learn spatio-temporal differential representations of videos. Experimental results demonstrate that our proposed scheme achieves satisfying perceptual performance at ultra-low bitrates and exhibits higher semantic fidelity in semantic analysis tasks such as human parsing. Code is available at: https://github.com/Chen8023/LPVC .