Effects of automatic alignment on speech translation metrics

Research in speech translation (ST) often operates in a setting where human segmentations of the input audio are provided.This simplifying assumption avoids the evaluation-time difficulty of aligning the translated outputs to their references for segment-level evaluation, but it also means that the systems are not evaluated as they will be used in production settings, where automatic audio segmentation is an unavoidable component.A tool, MwerSegmenter, exists for aligning ST output to references, but its behavior is noisy and not well understood.We address this with an investigation of the effects automatic alignment on metric correlation with system-level human judgments; that is, as a metrics task.Using the eleven language tasks from the WMT24 data, we merge each system's output at the domain level, align them to the references, compute metrics, and evaluate the correlation with the human system-level rankings.In addition to expanding analysis to many target languages, we also experiment with different subword models and with the generation of additional paraphrases.We find that automatic realignment has minimal effect on COMET-level system rankings, with accuracies still way above BLEU scores from manual segmentations.In the process, we also bring the community's attention to the source code for the tool, which we have updated, modernized, and realized as a Python module, mweralign. 1