Cross-Modal Haptic Generation for Emergency Rescue in Internet of Robotic Things

Robots equipped with multimodal sensing capabilities play an important role in the Internet of Robotic Things (IoRT), especially in emergency rescue. However, existing rescue robots pose challenges for human operators in achieving precise manipulation due to the absence of haptic signals. Additionally, limited and fluctuating bandwidth in emergency rescue renders current cross-modal haptic generation schemes ineffective. To overcome this dilemma, we propose a novel cross-modal haptic generation scheme that enhances scalability across diverse network conditions by leveraging correlations among audio-visual–haptic modalities. Specifically, we first propose an edge-device collaboration-based architecture that dynamically extracts semantics from audio-visual signals, tailored to the current network conditions, for haptic generation. This is achieved by predicting the network state at the edge and performing multimodal encoding and fusion at the device. Next, we design a scalable cross-modal haptic generation scheme that implements the optimal generation strategy based on received audio-visual semantics of different granularities, ensuring real-time acquisition of coarse-grained or fine-grained haptic feedback. Finally, numerical experimental results conducted on a multimodal dataset and a simulated emergency environment indicate that the proposed scheme reliably generates haptic signals in emergency rescue.