Prompt-to-Touch: Towards Enabling Automatic Haptic Effect Generation from Text Prompts Using Text-to-Audio Models

We introduce Prompt-to-Touch, a proof-of-concept of a multi-step pipeline that generates haptic effects based on the textual description of a haptic experience. Our pipeline first translates the haptic effect description to a sound effect description using our Foley-Interpreter component. It then uses a text-to-audio model to generate a sound effect from the sound description. Afterwards, the sound effect is converted into a perceivable haptic effect using our Dynamic-Audio-Processor . Finally, the haptic effect is post-processed to compensate for actuator-specific frequency response characteristics. We validate our concept in two preliminary human evaluation studies (n=20, n=10) and a technical analysis. Our results indicate that our pipeline can generate effects to enhance immersive multimedia experiences, abstract desktop/XR interactions, and social communication applications. They also still reveal significant potential for further improvement. Our pipeline could be used to develop future text-driven haptic design and automation tools. We provide open-source code to support future extensions.