A Comparative Study of ChatGPT and Papago Korean-to-English Translations of Dialogue from the Adaptation of Pachinko

The purpose of this study is to compare the performance of two Korean-to-English machine translations of Pachinko: ChatGPT-3.5 (ChatGPT) and Naver Papago (Papago). The drama dialogues were analyzed based on 590 pairs of samples translated using both ChatGPT and Papago to identify translation accuracy as well as lexical, grammatical, omission, and mechanical errors. These errors were further categorized into 11 subtype error patterns. The research found that ChatGPT produced 343 correct instances (58.1%) out of 590 attempts, while Papago achieved 223 correct translations (37.8%) for the same dataset. With regard to the four categories examined in this study, ChatGPT generated 190 lexical errors, 31 grammatical errors, 18 omissions, and 8 mechanical errors. In contrast, Papago yielded 301 lexical errors, 48 grammatical errors, 10 mechanical errors, and 8 omissions. Among the 11 error subtypes, the most frequent were incorrect word meaning errors, with 148 instances for ChatGPT and 226 for Papago, followed by pronoun errors, observed in 23 instances for ChatGPT and 40 for Papago. This comprehensive analysis of the primary error classifications reveals limitations in both ChatGPT and Papago. These findings underscore the need for future updates to enhance sensitivity to the unique aspects of the Korean language in both translation systems.