Evaluating the Accuracy and Naturalness of AI-Generated Translations Using BLEU and METEOR: A Comparison of ChatGPT, Gemini, Copilot, and DeepSeek

Artificial intelligence has caused a change in the area of machine translation within natural language processing. The focus on this research is on how these four AI models, ChatGPT, Gemini, Copilot and DeepSeek, can change English sentences into Indonesian. The analysis compares four types of artificial intelligence translators on their performance with ‘Pride and Prejudice’ by Jane Austen and verifies their results in Indonesian language. After the models did their translations, those texts were compared to those made by a professional translator for formal sentences, simple sentences and idioms. To complete the assessment, BLEU and METEOR scores were chosen and compared by looking at the output of the translation and the corresponding reference. For informal sentences, Gemini achieved the highest BLEU score (0.59), while both ChatGPT and Copilot reached the highest Recall Score (0.68). In translating formal sentences, ChatGPT performed best (BLEU 0.57, METEOR 0.59, Precision and Recall 0.66). DeepSeek had the highest Precision and Recall but scored lower on BLEU and METEOR. For Idiomatic Expressions, ChatGPT and DeepSeek led again, with scores around 0.52-0.53) for BLEU and METEOR. Gemini and Copilot showed the lowest performance in the formal and idiomatic category. After examining the translations, ChatGPT gives the clearest and most accurate results across all types of situations, mostly because informal English is challenging for the other apps. It will contribute to the ongoing research about language translation in AI and how it can be used or combined in settings where people jointly use language.