Cross-Modal Tactile Coding

Incorporating tactile feedback into traditional audio-visual multimedia services has gradually become the killer application in the Beyond 5G era. However, current state-of-the-art tactile coding approaches struggle to achieve extreme compression, thereby significantly impacting the transmission quality of visual streams due to resource competition. To address this challenge, this paper proposes an efficient cross-modal coding method that fully leverages semantic correlations between visual and tactile modalities. Specifically, we first construct a crossmodal coding architecture based on the predictive coding, enabling flexible and scalable bitrates to meet diverse compression requirements and adapt to dynamic network conditions. By introducing explainable surface semantics (e.g., friction, texture) as intermediates, we then associate visual modality with tactile modality to extract their potential correlations from the perspectives of multiple physical properties. Finally, we design a cross-modal feature fusion module through exploiting semantic correlations to further improve reconstruction quality of tactile signals, thereby reducing the bitrates required for tactile residuals and facilitating more efficient transmission. Experimental results demonstrate that the proposed tactile coding method achieves high bitrate compression, with almost no impact on visual stream quality.