Enhancing Real-Time Stream Processing with Apache Kafka by integrating Scaled-Hidden Markov Models

This research study explores the integration of Kafka, a distributed streaming platform, in the context of enhancing speech recognition systems utilizing Hidden Markov Models (HMMs). Speech recognition is a vital aspect of AI and Machine Learning which plays an important role in the interaction between humans and computers. Hidden Markov models (HMMs) are a popular choice for speech recognition systems due to sequential data modeling. In this paper, we conduct a comprehensive study integrating Big Data technologies like Kafka into Hidden Markov Model (HMM) based speech recognition, aiming to recognize spoken words from audio wave files and classify them into predefined word classes. The integration involves a multifaceted approach, employing Kafka for efficient data processing and communication between distinct system modules. To prevent underflow during HMM training, a Scaled HMM Module is employed. We address challenges posed by unseen data, by incorporating smoothing techniques such as add-delta smoothing which mitigates zero probabilities and improves the generalization of the model. We also experiment the use of pruning techniques which disregards the low-probability paths during the Viterbi algorithm, which results in improvement in the accuracy. By incorporating the integration of Big Data technologies like Kafka, along with advanced techniques such as pruning, smoothing, and scaling of Hidden Markov Models (HMMs), we delve into a comprehensive strategy for advancing speech recognition capabilities.