Adaptive Decoding for Efficient Automatic Speech Recognition

The latency and computation demand of End-to-end (E2E) automatic speech recognition (ASR) models hinder their deployment on lightweight devices. Despite there are many methods proposed for efficiency, the computational burden of the output layer with a large vocabulary is still a major challenge for decoders. In this paper, we propose an adaptive decoding method (ADD) to reduce the latency. Based on the vocal features of words like phonemes or speech units, we cluster the original vocabulary into small sets, allowing the model to inference more efficiently with a smaller search space. Experimental results demonstrate that our method significantly reduces the calculation FLOPs while maintaining performance. We also provide a deeper understanding of speech units from the perspective of phonemes.