Cross-Modal Transmission with Active Packet Loss and Restoration for Tactile Internet

Extended reality applications with haptic interaction over the Tactile Internet enable humans to perceive and manipulate remote environments in real time. However, due to the different requirements of each modality in terms of data rate, latency, and so on, as well as the complexity of characterizing the continuous relationship among modalities, when it comes to tele-manipulation with fine haptic experience in real time, how to effectively transmit these multi-modal streams while ensuring high-quality user experience when network congestion occurs remains a significant challenge. To address this challenge, this study proposes a cross-modal transmission strategy consisting of an active packet loss scheme and a cross-modal restoration procedure. The active packet loss scheme obtains tolerated packet loss probabilities by quantifying the quality of experience, along with predicting the restoration effect using a cross-modal reconstruction model that characterizes the relationship among modalities through real-time haptic rendering. When network congestion is detected, packets from the sender with a high tolerated packet loss probability are actively discarded with high priority. The receiver adopts the cross-modal reconstruction model from the active packet loss scheme to restore the lost signals through other modalities in real-time. We developed a visuo-haptic interaction testbed with 1 kHz force feedback to evaluate our cross-modal transmission strategy. Experimental results from both visual and haptic active packet loss tests show that the transmission gain of the overall mean opinion score exceeds 80 percent at a packet loss rate of 30 percent, indicating that our method enables a significant number of packets to be discarded, while maintaining a high-quality user experience, thereby alleviating network congestion.