Mitigating Hallucinations in Speech Recognition Systems for Noisy Data

Generative AI models have contributed significantly to advancements in various fields, with sequence-to-sequence transformers being particularly effective in Automatic Speech Recognition (ASR). However, these models often face the challenge of hallucinations, especially when processing noisy audio inputs, where noise here and throughout the study specifically refers to human noise (such as babble noise, a person mumbling or speaking gibberish). This research introduces a novel approach that utilizes context injection to mitigate hallucinations in ASR systems. The method involves minimal adjustments to the model pipeline, maintaining computational efficiency and avoiding increased model complexity. Evaluations conducted using the Whisper model, a transformer-based ASR framework, on a dataset of 30 noisy audio files demonstrated that the proposed solution significantly reduces hallucinations compared to conventional methods. These findings underscore the approach's potential to improve the reliability and accuracy of speech recognition in challenging environments.