A Framework for Large-Scale Parallel Corpus Evaluation: Ensemble Quality Estimation Models Versus Human Assessment

We developed a methodology and a framework for automatically evaluating and filtering largescale parallel corpora for neural machine translation (NMT).We applied six modern Quality Estimation (QE) models to score 55 million English-Ukrainian sentence pairs and conducted human evaluation on a stratified sample of 9,755 pairs.Using the obtained data, we ran a thorough statistical analysis to assess the performance of selected QE models and build linear, quadratic and beta regression models on the ensemble to estimate human quality judgments from automatic metrics.Our best ensemble model explained approximately 60% of the variance in expert ratings.We also found a non-linear relationship between automatic metrics and human quality perception, indicating that automatic metrics can be used to predict the human score.Our findings will facilitate further research in parallel corpus filtering and quality estimation and ultimately contribute to higher-quality NMT systems.We are releasing our framework, the evaluated corpus with quality scores, and the human evaluation dataset to support further research in this area.