Speech-Based Phonetic Transcript Metrics

Grapheme-to-phoneme (G2P) transducers play a critical role in the training of modern phonetic neural speech systems, e.g., automatic speech recognition and text-to-speech synthesis systems, as they generate phonetic transcripts to train these systems. The primary metric to evaluate the generated phonetic transcripts is the phonetic error rate (PER), calculated as the ratio of errors in the transcripts to the total number of phones in the human-annotated ground-truth (GT) transcript or pronunciation lexicon. However, human-annotated phonetic transcripts are rare, while pronunciation lexicons typically permit multiple context-dependent variants of each word (or, worse, omit words entirely); these resource scarcity problems hinder evaluation of high-quality context-dependent G2P transducers. In this work, we explore alternative speech-based phonetic transcript metrics in replacement of PER, leveraging existing corpora's parallel text and speech data. We propose to train speech-to-phone (S2P) and phone-to-speech (P2S) systems using the speech and phonetic transcripts to be evaluated and use the error rates of S2P or P2S generation as the measure for the degree of matching between speech and phonetic transcripts, and, hence, for the quality of phonetic transcripts. We conduct experiments to compare the proposed metrics with PER and demonstrate their strong correlation. In particular, we find S2P error rates exhibit a stronger correlation with PER than P2S error rates. Furthermore, we experiment on G2PU, a recently proposed acoustically augmented sentence-level G2P model, and illustrate how S2P error rates can guide the selection of the optimal setting, thereby eliminating its need for GT transcripts.