Benchmarking Large Language Models for Code Generation

Large Language Models (LLMs) have become increasingly popular for text-to-code generation, a task which involves converting natural language descriptions into code. While they have shown promising results, challenges remain with more complex problems and less common programming languages, highlighting the need for robust and systematic benchmarks. Without such benchmarks, there is a risk of misrepresenting LLMs capabilities and relying on tools that may not perform as expected. Existing benchmarks have several limitations, such as incomplete metrics, small datasets, a focus on a single programming language, and a lack of clear methodology. These shortcomings make it difficult to compare models effectively and limit the benchmarks' ability to expand to cover important aspects such as dependability and security. This work introduces a systematic benchmarking framework to evaluate LLMs on text-to-code tasks. The framework considers the need for diverse problems, multiple evaluation metrics, and supports setups like few-shot learning, aiming to provide meaningful insights into LLMs performance and their applicability in real-world programming scenarios.