Contrastive Learning for Task-Independent SpeechLLM-Pretraining

Large language models (LLMs) excel in natural language processing but adapting these LLMs to speech processing tasks efficiently is not straightforward.Direct task-specific fine-tuning is limited by overfitting risks, data requirements, and computational costs.To address these challenges, we propose a scalable, two-stage training approach: (1) A taskindependent speech pretraining stage using contrastive learning to align text and speech representations over all layers, followed by (2) a taskspecific fine-tuning stage requiring minimal data.This approach outperforms traditional ASR pretraining and enables the model to surpass models specialized on speech translation and question answering while being trained on only 10% of the task-specific data.