Duration-Aware Phone Embedding Upsampling for Open Vocabulary Keyword Spotting

Open Vocabulary Keyword Spotting (KWS) gained significant interest due to the ability to detect custom keywords. Still, it is a very challenging field to develop, as it has to overcome domain mismatch and rate mismatch. Existing methods solve domain mismatch by bringing the acoustic and text keyword embeddings into shared latent space. Still, they face challenges such as rate mismatch where speech frames and text keyword phonemes have different temporal resolutions, and struggle to find agreement between the modalities. To address these, we propose a method for open vocabulary keyword spotting that matches the rate. It consists of an Acoustic encoder, Text encoder, Duration estimator, Upsampler, Localizer, and Detector modules. The acoustic and text encoders map inputs into a shared latent space, enabling effective cross-modal representation. The duration estimator takes the phoneme embeddings and predicts their durations. Then the Upsampler interpolates these embeddings to an acoustic rate based on the predicted durations. The resultant upsampled embeddings aligned with acoustic frame embeddings using a multi-head cross-attention mechanism. Following this, the localizer identifies the keyword's location, and the detector predicts its existence. Our experimental results demonstrate that this approach performs significantly better than the baseline and effectively addresses key challenges in KWS, including domain and rate mismatches.