Improving Spoken Semantic Parsing using Synthetic Data from Large Generative Models

Spoken semantic parsing (SSP) involves generating machine-comprehensible parses from input speech.Training robust models for existing application domains represented in training data or extending to new domains requires corresponding triplets of speech-transcript-semantic parse data, which are expensive to obtain.In this paper, we address this challenge by examining methods that can use transcript-semantic parse data (unpaired text) without the corresponding speech.First, when unpaired text is drawn from existing textual corpora, Joint Audio Text (JAT) and Text-to-Speech (TTS) are compared as ways to generate speech representations for unpaired text.Experiments on the STOP dataset show that unpaired text from existing and new domains improves performance by 2% and 30% in absolute Exact Match (EM) respectively.Second, we consider the setting when unpaired text is not available in existing textual corpora.We propose prompting Large Language Models (LLMs) to generate unpaired text for existing and new domains.Experiments show that examples and words that cooccur with intents can be used to generate unpaired text with Llama 2.0.Using the generated text with JAT and TTS for spoken semantic parsing improves EM on STOP by 1.4% and 2.6% absolute for existing and new domains, respectively.