Comparing the Translation Quality of Large Language Models in Legal Texts: A Quantitative Analysis

With the rapid development of artificial intelligence and large language models (LLMs), the potential for automatic legal translation has garnered significant interest across the globe. This study evaluates the performance of four mainstream LLMs — ChatGPT 40, Gemini 2.0 Pro Experimental, DeepSeek R1, and GLM Plus - in Chinese-to-English legal translation, using the China's Civil Code as parallel corpus. By employing quantitative evaluation metrics including BLEU, chrF++, and BERTScore, the study provides a comprehensive analysis of the models' application efficacy in translating legal texts. The results show significant differences in translation performance, with Gemini 2.0 Pro Experimental outperforming the other models, indicating its superior ability in handling legal terminology and complex sentence structures. In contrast, DeepSeek R1 demonstrated the weakest performance, reflecting its limitations in translating intricate legal texts. GLM Plus and ChatGPT 40 exhibited similar performances, but there were noticeable differences in detail handling and stability. These findings highlight the potential and challenges of using LLMs for legal translation, providing valuable insights for future improvements in the field.