CRoss-MODAL Communications For Holographic Video Streaming

Holographic video, coupled with spatial audio and haptic feedback, is gradually emerging as an enabling medium to deliver interactive and immersive experiences. However, streaming holographic video over wireless networks remains a significant challenge due to high bandwidth demand, resource competition among different modalities, and the dynamic nature of network conditions. To tackle these challenges, we introduce the cross-modal communications paradigm in this work, which exploits potential correlations among video, audio, and haptic modalities to enhance holographic video streaming from both encoding and transmission perspectives. Specifically, we first propose a cross-modal visual saliency prediction method for holographic video compression, leveraging spatiotemporal information provided by spatial audio and haptic feedback to improve prediction accuracy. Then, by exploring semantic correlations between video and haptic modalities, we establish a perceptionIossless haptic coding architecture for extreme haptic compression to alleviate resource competition with holographic video streams. Finally, based on transmission priority settings of video tiles guided by audio and haptic streams, we develop an efficient holographic video rate adaption scheme under time-varying network conditions to consistently ensure immersive experiences. Numerical results validate the benefits of cross-modal communications for holographic video streaming.