SDL: Spoken Dual Learning for Joint Optimization of SLU and SLG

This paper presents a novel spoken dual learning (SDL) frame-work aimed at the concurrent training of spoken language understanding (SLU) and generation (SLG) models. The central proposition is to consider SLU and SLG as conditional text generation tasks steered by prompts, thereby offering consistent goals and context-aware direction. SDL navigates the inherent dual task constraints through an innovative transformation technique utilizing noisy transcripts and intents. The blueprint of SDL incorporates high-profile, pre-trained speech models such as Wav2vec2 or Whisper, serving as the backbone. Extensive experimentation on datasets of varying sizes (small, medium, and large) provides empirical evidence demonstrating that SDL exhibits higher performance levels compared to models that are independently trained, as the cycle consistency of mapping be-tween speech, semantics, and text acts as effective regularization.