On the Robust Approximation of ASR Metrics

Recent advances in speech foundation models are largely driven by scaling both model size and data, enabling them to perform a wide range of tasks, including speech recognition.Traditionally, ASR models are evaluated using metrics like Word Error Rate (WER) and Character Error Rate (CER), which depend on ground truth labels.As a result of limited labeled data from diverse domains and testing conditions, the true generalization capabilities of these models beyond standard benchmarks remain unclear.Moreover, labeling data is both costly and time-consuming.To address this, we propose a novel label-free approach for approximating ASR performance metrics, eliminating the need for ground truth labels.Our method utilizes multimodal embeddings in a unified space for speech and transcription representations, combined with a high-quality proxy model to compute proxy metrics.These features are used to train a regression model to predict key ASR metrics like Word Error Rate (WER) and Character Error Rate (CER).We experiment with over 40 models across 14 datasets representing both standard and in-thewild testing conditions.Our results show that we approximate the metrics within a singledigit absolute difference across all experimental configurations, outperforming the most recent baseline by more than 50%.