Evaluating Large Language Models: A Systematic Survey of Benchmarks, Contamination, and Human Evaluation

Large language models are now deployed in settings where their failures carry real costs, yet the methods used to judge them have not kept pace with their capabilities. A single benchmark score conceals prompt sensitivity, training-data contamination, evaluator bias, and the gap between automated metrics and human judgment. This paper presents a systematic survey of the evaluation science of large language models. It organizes evaluation into capability benchmarks, task-specific suites, human evaluation, and model-based judgment; reviews the measurement foundations of automated metrics and statistical comparison; examines the central threats to validity — data contamination, benchmark saturation, prompt and decoding sensitivity, and judge bias; and synthesizes emerging practices such as contamination-aware benchmark design, dynamic and private evaluation, and arena-style human preference ranking. The survey concludes that trustworthy evaluation is no longer a matter of running one more benchmark: it requires triangulation across independent evidence types, transparent reporting of evaluation conditions, and benchmarks designed to resist the very memorization they are meant to detect.