A Multi-Modal Based Gesture-to-Speech Conversion System

Gesture to speech conversion is a vital technology for enhancing communication accessibility for individuals with speech impairments or those operating in environments where verbal communication is challenging. This paper proposes a novel approach for real-time gesture to speech conversion by using the Media Pipe Holistic Model, a state-of-the-art framework for human pose estimation, facial landmark detection, and hand tracking. The system utilizes the comprehensive information extracted from the Media Pipe Holistic Model, including body poses, facial expressions, and hand gestures, to infer the user’s intended message and convert it into synthesized speech output. The proposed system begins by capturing input gestures through a camera feed, which are then processed using the Media Pipe Holistic Model to extract relevant body, facial, and hand features. These features are subsequently fed into a gesture model ,employing python code, to interpret the gestures and map them to predefined speech commands or textual representations. Additionally, facial expressions and hand gestures are analyzed to further enhance the accuracy and naturalness of the synthesized speech output. To facilitate real-time performance and usability, the system is optimized for efficient inference on various computing platform. The utilization of the Media Pipe Holistic Model enables seamless integration with existing applications and frameworks. Overall, the proposed approach showcases the capabilities of using advanced computer vision techniques, such as the Media Pipe Holistic Model, to develop innovative solutions for gesture-based applications in assistive technology, human computer interaction, and augmented reality interfaces recognition.