Evaluating Transformer-Based Architectures for Simultaneous Audio Speech Transcription and Background Audio Captioning

This study evaluates transformer-based models for simultaneous speech transcription and background audio captioning in mixed audio scenarios. Using Whisper for speech and Prompteus for environmental sounds, the models are tested on the Clotho Voice dataset, which combines Portuguese speech (Common Voice 5.1) and environmental audio (Clotho 2.1). Results using WER and FENSE metrics show that each model performs well in its domain but degrades with overlapping signals. Whisper is robust to moderate noise, while Prompteus struggles with dominant speech. The findings highlight the need for hybrid approaches to enable reliable, context-aware audio processing in complex environments.