CABench: Evaluating Constraint-Aware Pragmatism and Algorithmic Over-Engineering in LLM Code Generation

Existing code generation benchmarks—including LiveCodeBench, EffiBench, Mercury, and AlgoTune—evaluate functional correctness or asymptotic scalability on large inputs ($N \ge 10^5$), implicitly rewarding heavyweight data structures regardless of problem bounds. When competitive programming (ICPC) constraints are tightly bounded ($N \le 20$), large language models (LLMs) frequently over-engineer solutions with Segment Trees, priority queues, or dynamic programming (DP) tables that suffer from heap allocation and cache overhead rather than emitting pragmatic, cache-friendly brute-force or $O(2^N)$ search. We introduce CABench, a constraint-mutated benchmark of 60 held-out ICPC archetypes across three constraint tiers (Tier S, Tier M, Tier L; 180 evaluation instances) paired with formal metrics: Constraint-Relative Speedup (CRS), Over-Engineering Penalty Index (OEPI), and Pragmatic Efficiency Score (PES). An a priori power analysis shows that $n=60$ paired archetypes per tier halves the minimum detectable effect ($d_z = 0.368$ at 80% power) relative to a 15-archetype pilot ($d_z = 0.778$). Across 8 models spanning closed frontier, open-weight APIs, and local 7B–8B models (1,440 evaluated C++20 programs inside an isolated rootless Podman harness), Claude Haiku 5.5 achieves 100.0% Tier S pass@1 ($\text{PES}=0.733$), GPT-6 Luna reaches 96.7% ($\text{PES}=0.698$), and GLM-5.3-Flash achieves the lowest frontier structural bloat ($\text{OEPI}=0.255$ at Tier S, $0.364$ at Tier L; $p_{\text{Holm}} = 0.0067$ vs. Claude Haiku 5.5). Finally, we show that both zero-shot constraint-aware prompting on Claude Haiku 5.5 ($\text{OEPI}\ 0.359 \to 0.554$, $p = 0.0040$) and 1-epoch QLoRA SFT on Qwen3-8B fail to jointly optimize pragmatism and accuracy (SFT cuts mutually passed OEPI across tiers, $p \le 0.0429$, but degrades Tier M/L pass@1, $p_{\text{Holm}} \le 0.0190$). In contrast, inference-time Best-of-4 PES-guided reranking on GLM-5.3-Flash Tier S simultaneously raises pass rate ($86.7\% \to 98.3\%$, $p = 0.0156$), cuts mutually passed OEPI ($0.255 \to 0.210$, $p = 0.0051$), and achieves $\text{PES} = 0.828$ ($p = 2.04 \times 10^{-7}$).

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-21
DOI
https://doi.org/10.5281/zenodo.24420443
Primary Topic
Software Engineering Research
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

CABench: Evaluating Constraint-Aware Pragmatism and Algorithmic Over-Engineering in LLM Code Generation

Juan S. Santillana
Zenodo (CERN European Organization for Nuclear Research)
Software Engineering Research
preprint

CABench: Evaluating Constraint-Aware Pragmatism and Algorithmic Over-Engineering in LLM Code Generation

Juan S. Santillana
preprint en

Abstract

Existing code generation benchmarks—including LiveCodeBench, EffiBench, Mercury, and AlgoTune—evaluate functional correctness or asymptotic scalability on large inputs ($N \ge 10^5$), implicitly rewarding heavyweight data structures regardless of problem bounds. When competitive programming (ICPC) constraints are tightly bounded ($N \le 20$), large language models (LLMs) frequently over-engineer solutions with Segment Trees, priority queues, or dynamic programming (DP) tables that suffer from heap allocation and cache overhead rather than emitting pragmatic, cache-friendly brute-force or $O(2^N)$ search. We introduce CABench, a constraint-mutated benchmark of 60 held-out ICPC archetypes across three constraint tiers (Tier S, Tier M, Tier L; 180 evaluation instances) paired with formal metrics: Constraint-Relative Speedup (CRS), Over-Engineering Penalty Index (OEPI), and Pragmatic Efficiency Score (PES). An a priori power analysis shows that $n=60$ paired archetypes per tier halves the minimum detectable effect ($d_z = 0.368$ at 80% power) relative to a 15-archetype pilot ($d_z = 0.778$). Across 8 models spanning closed frontier, open-weight APIs, and local 7B–8B models (1,440 evaluated C++20 programs inside an isolated rootless Podman harness), Claude Haiku 5.5 achieves 100.0% Tier S pass@1 ($\text{PES}=0.733$), GPT-6 Luna reaches 96.7% ($\text{PES}=0.698$), and GLM-5.3-Flash achieves the lowest frontier structural bloat ($\text{OEPI}=0.255$ at Tier S, $0.364$ at Tier L; $p_{\text{Holm}} = 0.0067$ vs. Claude Haiku 5.5). Finally, we show that both zero-shot constraint-aware prompting on Claude Haiku 5.5 ($\text{OEPI}\ 0.359 \to 0.554$, $p = 0.0040$) and 1-epoch QLoRA SFT on Qwen3-8B fail to jointly optimize pragmatism and accuracy (SFT cuts mutually passed OEPI across tiers, $p \le 0.0429$, but degrades Tier M/L pass@1, $p_{\text{Holm}} \le 0.0190$). In contrast, inference-time Best-of-4 PES-guided reranking on GLM-5.3-Flash Tier S simultaneously raises pass rate ($86.7\% \to 98.3\%$, $p = 0.0156$), cuts mutually passed OEPI ($0.255 \to 0.210$, $p = 0.0051$), and achieves $\text{PES} = 0.828$ ($p = 2.04 \times 10^{-7}$).

Zenodo (CERN European Organization for Nuclear Research)
Software Engineering Research
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon
CABench: Evaluating Constraint-Aware Pragmatism and Algorithmic Over-Engineering in LLM Code Generation — Juan S. Santillana · Zenodo (CERN European Organization for Nuclear Research) (2026) | TGRS Research Map | TGRS