Multi-dimensional Texture Haptic Cross-modal Generation and Display Method based on Bi-Mamba Network

The haptic perception of texture has significant multi-dimensional (MD) characteristics, mainly composed of key attributes such as roughness, friction, and hardness. These dimensions not only reflect the microstructure of object surfaces, but also serve as the foundation for achieving real haptic feedback in virtual interactions. The integration of MD perception enhances the consistency and immersion of haptic rendering, especially in dynamic interactions that integrate user action information. However, existing modeling methods often focus on a single dimension, which is difficult to meet the demand for high immersion and adaptability. Therefore, we proposed an end-to-end MD texture haptic rendering model. This model takes the Mamba encoder as the core, integrates texture images and the real-time action information of users, and jointly predicts high-quality acceleration and friction signals. To train and validate the model, a set of equipment for collecting sliding speed and normal pressure was designed. Data interacting with 70 real textures were collected, and a training set was constructed in combination with the SENS3 dataset. In tests conducted across four distinct scenarios, our model exhibited robust signal reconstruction capabilities (RMSE is 0.032), outperforming the baseline model in overall performance. And it demonstrated strong generalization abilities for common textures such as leather, fabric, and wood. Additionally, the delay of the haptic display system controlled between 32∼38 ms, below the 40 ms haptic perception threshold. Finally, we conducted two user experiments. The results indicate that the rendering model that integrates roughness and friction feedback significantly improves the accuracy of users in distinguishing between real and virtual textures. The proposed method achieved the highest perceptual average similarity score to date (6.99), effectively enhancing the user’s realistic experience with virtual textures. Our code can be available at https://github.com/CdpLab/MDTextHaptDisplay.