Improving weakly supervised ASR from TV subtitles via refinement and adversarial training

Automatic speech recognition (ASR) systems typically require large volumes of high-quality transcribed speech. In low-resource settings, where accurate verbatim transcripts are scarce, weakly supervised (WS) sources provide a practical alternative. In this work, we use the audio, subtitle texts, and subtitle timestamps from 16 television programs broadcast by the Vlaamse Radio- en Televisieomroeporganisatie (VRT, Flemish Radio and Television Broadcasting Organisation) as WS resources for ASR training. We examine the value of subtitles by comparing them with pseudo transcripts generated by existing ASR models, which are commonly used in WS ASR. Since TV subtitles are often imperfectly aligned with speech, we first propose a subtitle refinement pipeline that uses Whisper-based pseudo transcriptions, forced alignment, and subtitle-specific processing to produce better-aligned training pairs. We then introduce a more flexible end-to-end alternative based on adversarial training, which models text predictions at the distribution level and reduces the reliance on precise text–speech alignment. Experiments with different ASR backbones show that training on refined subtitles substantially improves recognition performance over raw subtitles and direct pseudo-transcript supervision. The adversarial approach achieves competitive overall performance while offering a simpler training framework for more weakly aligned settings.