Real-Time Semantic Video Communication with Temporally Consistent And Controllable Diffusion Models

This paper introduces CVSC, a real-time-enabled and temporally consistent semantic video communication approach. We minimize the denoising steps of the diffusion model by using the most recent frame and motion information, enabling real-time-capable semantic video synthesis. To ensure temporal consistency, we employ windowed temporal cross-attention. While our objective evaluation highlights the limitations of existing metrics for generative models in semantic video communication, subjective evaluations demonstrate the superiority of our approach in terms of human preference at extremely low bit rates. (< 0.006 bpp).