Usporedno istraživanje upotrebe ChatGPT-ja i Google prevoditelja za prevođenje zakonodavnih tekstova

Large language models, such as ChatGPT, are being rapidly developed and used across a wide range of fields, including translation. Translators of legislative texts have a particular interest in automating their workflow, given the repetitive nature of the material they work with. As of yet, little research has been done on ChatGPT’s performance with low-resource languages such as Croatian. The aim of this thesis is to compare the quality of ChatGPT’s translation with Google Translate in order to establish whether it is better suited for translating legislative texts from English into Croatian than the more traditional machine translation (MT) system. In order to test this, Decision (EU) 2022/1628 was machine-translated with ChatGPT-3.5 and Google Translate, after which both outputs were post-edited. The target texts were automatically evaluated via multiple metrics, including BLEU, chrF, BERTScore, BLEURT, TER, and COMET. Additionally, the thesis presents a detailed error analysis conducted during the post-editing process, with comments on the most frequent errors found in both translations. The results of the automatic evaluation suggest that Google Translate is currently slightly better at translating legislative texts than ChatGPT-3.5. Errors were found in ChatGPT’s output that were not present in GT’s translation and vice versa, with both systems failing to adequately translate certain EU legal terminology. Any generalizations that can be drawn from the findings are limited, as the study would greatly benefit from a larger sample size, other language combinations, as well as the inclusion of non-biased human evaluators, all of which should be addressed in future research.