WhisperFlow: speech foundation models in real time

Speech foundation models, such as OpenAI's Whisper, become the state of the art in speech understanding due to their strong accuracy and generalizability. Yet, their applications are mostly limited to processing pre-recorded speech, whereas processing of streaming speech, in particular doing it efficiently, remains rudimentary. Behind this inefficiency are multiple fundamental reasons: (1) speech foundation models are trained to process long, fixed-length voice inputs (often 30 seconds); (2) encoding each voice input requires encoding as many as 1,500 tokens with tens of transformer layers; (3) decoding each output entails an irregular, complex beam search. As such, streaming speech processing on resource-constrained client devices is more expensive than other AI tasks, e.g., text generation.