This Model Are Sick: A Comparative Analysis of In and Out-of-Domain Datasets in NMT-Driven Video Game Localisations

Latterly, although major advancements have been achieved in the field of neural machine translation, its employment within the video game localisation sector has been sorely neglected in the process. As such, owing to the complexity, a smattering of available resources, and creativity necessitated within the sector, research, as well as any supporting literature on the matter, have been sparse and difficult to obtain. The purpose of this thesis is to serve as an advent into the research of the neural machine translation approach to video game localisations. Given the dearth of experimentation and research within the sector, this project attempts to bring forth attention to its necessities, obstacles, and potential implementation. Optimally, our analysis would at once opt to observe the performance of more advanced, high-parameter systems; however such an approach demands access to vast amounts of computational and monetary resources. Instead, our efforts are concerned with evaluating the performance of two from-scratch trained and two fine-tuned model architectures of a reasonable size. Attesting to our model performance via three different evaluation metrics for machine translation, we observe that the performance on the from-scratch trained models is, unsurprisingly, quite poor. The accuracy of our fine-tuned models, particularly the MarianMT architecture, shows promising results pertaining to its ability to at times adhere to the domain at hand, and offer slightly better translations than its non fine-tuned counterpart. This suggests that, given enough data, these models could display a high enough rate of in-domain adaptability to be used in a post-editing workflow. Additionally, we observe that our best-performing fine-tuned model had out-performed the publicly available Google Translate system across both COMET metrics, although its performance is still some ways away of the skyline achieved by large language models such as DeepSeek. Lastly, we discover that certain challenges that plague the field of machine translation, such as gender bias and a high computational overhead, are blatantly present within the video game sector as well.