Assessing and Advancing Benchmarks for Evaluating Large Language Models in Software Engineering Tasks

Large Language Models (LLMs) are gaining increasing popularity in Software Engineering (SE) due to their unprecedented performance across various applications. These models are increasingly being utilized for a range of SE tasks, including requirements engineering and design, code analysis and generation, software maintenance, and quality assurance. As LLMs become more integral to SE, evaluating their effectiveness is crucial for understanding their potential in this field. In recent years, substantial efforts have been made to assess LLM performance in various SE tasks, resulting in the creation of several benchmarks tailored to this purpose. This article offers a thorough review of 291 benchmarks, addressing three main aspects: what benchmarks are available , how benchmarks are constructed , and the future outlook for these benchmarks . We begin by examining SE tasks such as requirements engineering and design, coding assistant, software testing, AIOps, software maintenance, and quality management. We then analyze the benchmarks and their development processes, highlighting the limitations of existing benchmarks. Additionally, we discuss the successes and failures of LLMs in different software tasks and explore future opportunities and challenges for SE-related benchmarks. We aim to provide a comprehensive overview of benchmark research in SE and offer insights to support the creation of more effective evaluation tools.

Authors

Institutions

Publication Details

Journal
ACM Transactions on Software Engineering and Methodology
Published
2026-11-18
DOI
https://doi.org/10.1145/3786771
Citations
60
Primary Topic
Software Engineering Research
Type
article
Field-Weighted Citation Impact
110.75
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Assessing and Advancing Benchmarks for Evaluating Large Language Models in Software Engineering Tasks

Junwei Zhang, Xin Xia, Feifei Niu, David Lo et al.
60 citations
ACM Transactions on Software Engineering and Methodology
Software Engineering Research
110.75
article

Assessing and Advancing Benchmarks for Evaluating Large Language Models in Software Engineering Tasks

Junwei Zhang, Xin Xia, Feifei Niu, David Lo, Xin Zhou, Xi Xia, Junda He, X X Hu, Xin Zhou, Junkai Chen, Junkai Chen
article en
60 citations

Abstract

Large Language Models (LLMs) are gaining increasing popularity in Software Engineering (SE) due to their unprecedented performance across various applications. These models are increasingly being utilized for a range of SE tasks, including requirements engineering and design, code analysis and generation, software maintenance, and quality assurance. As LLMs become more integral to SE, evaluating their effectiveness is crucial for understanding their potential in this field. In recent years, substantial efforts have been made to assess LLM performance in various SE tasks, resulting in the creation of several benchmarks tailored to this purpose. This article offers a thorough review of 291 benchmarks, addressing three main aspects: what benchmarks are available , how benchmarks are constructed , and the future outlook for these benchmarks . We begin by examining SE tasks such as requirements engineering and design, coding assistant, software testing, AIOps, software maintenance, and quality management. We then analyze the benchmarks and their development processes, highlighting the limitations of existing benchmarks. Additionally, we discuss the successes and failures of LLMs in different software tasks and explore future opportunities and challenges for SE-related benchmarks. We aim to provide a comprehensive overview of benchmark research in SE and offer insights to support the creation of more effective evaluation tools.

ACM Transactions on Software Engineering and MethodologyVol. 35(12)
University of Ottawa (CA), Zhejiang University (CN), Singapore Management University (SG)
Openalex Percentile: Top 0%
Software Engineering Research
110.75
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon