BLEU: Bilingual Evaluation Understudy BLEU Score for Machine Translation

The BLEU (Bilingual Evaluation Understudy) metric has served as a cornerstone in the automatic evaluation of machine translation and natural language generation systems for over two decades. Despite its widespread adoption and influence, misunderstandings about its computation, interpretation, and appropriate usage remain common. This paper provides a detailed and systematic examination of BLEU, covering its original formulation, modified precision mechanism, brevity penalty, smoothing strategies, and the role of multiple reference translations. The discussion includes illustrative examples and edge cases, analyzes the metric’s strengths and weaknesses in both traditional MT and modern LLM-based generation tasks, and situates BLEU in the context of contemporary evaluation metrics. By clarifying the assumptions, limitations, and implementation nuances of BLEU, this work aims to support robust evaluation practices across a wide range of language generation scenarios.