Revisiting SWE-Bench: On the Importance of Data Quality for LLM-Based Code Models

The use of Large Language Models (LLMs) for code generation has emerged as a rapidly growing field, gaining substantial traction within software engineering. However, ensuring the reliability and accuracy of generated code requires robust evaluation frameworks. To address this gap, Carlos et al. introduced the SWE-bench dataset, which consists of 2,294 GitHub issues paired with their corresponding pull requests, collected from 12 prominent Python repositories. This dataset has become a key benchmark for evaluating code generation models, with resolution rates prominently featured on the SWE-bench leaderboard. Despite its widespread adoption, the dataset has yet to undergo a systematic reliability assessment. Motivated by this gap, we conducted the first empirical study aimed at evaluating the reliability of the SWE-Bench dataset to ensure it provides meaningful and realistic model evaluations. We centered our analysis on the highest-performing model reported on the leaderboard at the time of the study: SWE-Agent + GPT-4. A thorough investigation was conducted by comparing the model-generated patches with the corresponding pull requests from the dataset. Our findings revealed two key issues: (1) 32.67% of successful cases were influenced by solution leakage, and (2) 31.08% succeeded due to weak test cases. When these problematic instances were excluded, the resolution rate of SWE-Agent + GPT-4 dropped from 12.47% to 3.97%.