Effective Multilingual and Mixed-lingual DSR System for Healthcare Application in Indian Languages

Voice-controlled technologies like Google Assistant, Siri, Alexa, heavily rely on accurate speech recognition. But in a multilinguistic country like China, India, and Australia, where people speak more than one language, existing digit speech recognition (DSR) systems often fall short. They struggle to handle the complexities of mixed languages. Acknowledging the existing problem, this work proposes a multilingual and mixed-lingual DSR system. The system utilizes the OpenAI Whisper model and Convolutional Neural Network (CNN) to recognize the language and the digit spoken by the user respectively. To train the CNN model, a comprehensive database of spoken digits (0-9) was built in four Indian languages: Marathi, Gujarati, Bengali, and Malayalam. The extracted spectrographic features from the recordings are used by CNN to accurately recognize the spoken digits. The suggested DSR model yielded an accuracy ranging from 94.8% to 100% for different languages, while the mixed-lingual DSR model achieved 99% accuracy. These astonishing results are obtained due to several key aspects such as: (i) Effective data preprocessing, (ii) Batch Normalization in CNN, (iii) 5-Fold cross validation in training. The DSR model is used for the healthcare industry to help patients request services using voice commands, i.e. digits 0-9. Key benefits of the healthcare application include: (i) Enhanced Patient Experience, (ii) Operational Efficiency, (iii) Increased Accessibility.