Maintaining Prosodic Consistency in Automatic Dubbing for Better Isochrony

One of the major requirements of automatic dubbing, as an extension of speech-to-speech translation, is isochrony. It refers to the fact that dubbed speech should line up in time with the original speech, matching the phrase-pause arrangement of the source utterance. To achieve this, we explore and compare two alternative prosodic alignment modeling techniques: Prosodic-only and joint Prosodic-MT. Based on the original speech phrase-pause structure, the former aims to predict the optimal position of pause markers in the target text sequence, while the latter implicitly incorporates prosodic alignment within the translation process to generate a suitable translation for dubbing. Empirical results on two publicly available datasets demonstrate the effectiveness of the proposed approaches in terms of both translation quality and syllables-based isochrony metrics.