CABench: Evaluating Constraint-Aware Pragmatism and Algorithmic Over-Engineering in LLM Code Generation
Existing code generation benchmarks—including LiveCodeBench, EffiBench, Mercury, and AlgoTune—evaluate functional correctness or asymptotic scalability on large inputs ($N \ge 10^5$), implicitly rewarding heavyweight data structures regardless of problem bounds. When competitive programming (ICPC) constraints are tightly bounded ($N \le 20$), large language models (LLMs) frequently over-engineer solutions with Segment Trees, priority queues, or dynamic programming (DP) tables that suffer from heap allocation and cache overhead rather than emitting pragmatic, cache-friendly brute-force or $O(2^N)$ search. We introduce CABench, a constraint-mutated benchmark of 60 held-out ICPC archetypes across three constraint tiers (Tier S, Tier M, Tier L; 180 evaluation instances) paired with formal metrics: Constraint-Relative Speedup (CRS), Over-Engineering Penalty Index (OEPI), and Pragmatic Efficiency Score (PES). An a priori power analysis shows that $n=60$ paired archetypes per tier halves the minimum detectable effect ($d_z = 0.368$ at 80% power) relative to a 15-archetype pilot ($d_z = 0.778$). Across 8 models spanning closed frontier, open-weight APIs, and local 7B–8B models (1,440 evaluated C++20 programs inside an isolated rootless Podman harness), Claude Haiku 5.5 achieves 100.0% Tier S pass@1 ($\text{PES}=0.733$), GPT-6 Luna reaches 96.7% ($\text{PES}=0.698$), and GLM-5.3-Flash achieves the lowest frontier structural bloat ($\text{OEPI}=0.255$ at Tier S, $0.364$ at Tier L; $p_{\text{Holm}} = 0.0067$ vs. Claude Haiku 5.5). Finally, we show that both zero-shot constraint-aware prompting on Claude Haiku 5.5 ($\text{OEPI}\ 0.359 \to 0.554$, $p = 0.0040$) and 1-epoch QLoRA SFT on Qwen3-8B fail to jointly optimize pragmatism and accuracy (SFT cuts mutually passed OEPI across tiers, $p \le 0.0429$, but degrades Tier M/L pass@1, $p_{\text{Holm}} \le 0.0190$). In contrast, inference-time Best-of-4 PES-guided reranking on GLM-5.3-Flash Tier S simultaneously raises pass rate ($86.7\% \to 98.3\%$, $p = 0.0156$), cuts mutually passed OEPI ($0.255 \to 0.210$, $p = 0.0051$), and achieves $\text{PES} = 0.828$ ($p = 2.04 \times 10^{-7}$).
Authors
- Juan S. Santillana
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-21
- DOI
- https://doi.org/10.5281/zenodo.24420443
- Primary Topic
- Software Engineering Research
- Type
- preprint