A Comparative Analysis of Human and Machine Translation Quality

A common question raised by both translators and Machine Translation developers is Will Machine Translation (MT) ever attain the level of Human Translation (HT) quality? Researchers have argued that MT will likely never replace HT completely because MT systems cannot anticipate all the specifications for possible translation projects (Melby & Hague, 2019) and because MT systems lack cultural understanding (Latief et al., 2020). Researchers have also argued that the use of machines should be focused on augmenting humans, rather than trying to imitate them (Brynjolfsson, 2022). The purpose of this study is to determine how the quality of MT differs from the quality of HT. The HT data for this study comes from a database of translation certification exams and includes the evaluations of the exams. The MT data comes from using the sources texts shown in the database and using GPT-4 and Microsoft Translator to translate them. The quality of the translations is determined quantitatively with a quality score and qualitatively by analyzing the different types of errors prevalent in both types of translation. This study uses the Multidimensional Quality Metrics (MQM) framework to classify errors. This thesis focuses on data from four language pairs: Spanish to English, English to Spanish, Portuguese to English, and English to Portuguese. In this study, it is observed that, although there is not a significant difference in the quantitative quality scores between HT and MT, the qualitative information about the MQM error types provides insight into the differences between HT and MT for translating to and from English, Spanish, and Portuguese. This insight establishes the strengths and weaknesses of the two types of translations for these language pairs and also informs the training of human translators and the direction of quality improvement of machine translation.