WHEN CORRECT CODE IS NOT ENOUGH: ASSESSING THE PRODUCTION READINESS OF LLMGENERATED SOFTWARE

Large language models (LLMs) increasingly generate executable software, but functional correctness alone does not establish production readiness. This study evaluates GPT-5.6 Sol, Terra, and Luna across 15 algorithmic and production-oriented tasks using five generations per model-task pair, yielding 225 generated programs plus 15 fixed references. The layered testbed combines nominal and hidden-edge-case checks, moderate and severe production-like stress, metamorphic validation, reference-normalized runtime and memory, structural quality, static analysis, implementation-strategy inspection, and generation cost. All 800 functional and 368 applicable robustness checks passed, while production-stress success fell to 95.16% for Sol, 95.60% for Terra, and 93.38% for Luna. Cross-model stress differences were not significant, but generation cost differed strongly. A Quick Sort case showed more than a 500x runtime spread among behaviorally correct implementations. The results show that production-oriented evaluation must extend beyond pass/fail correctness to include stress behavior, resources, implementation strategy, and cost

Authors

Publication Details

Journal
International Journal of Software Engineering & Applications
Published
2026-11-04
DOI
https://doi.org/10.5121/ijsea.2026.17601
Primary Topic
Software Engineering Research
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

WHEN CORRECT CODE IS NOT ENOUGH: ASSESSING THE PRODUCTION READINESS OF LLMGENERATED SOFTWARE

Ramesh Venkatachalapathy
International Journal of Software Engineering & Applications
Software Engineering Research
article

WHEN CORRECT CODE IS NOT ENOUGH: ASSESSING THE PRODUCTION READINESS OF LLMGENERATED SOFTWARE

Ramesh Venkatachalapathy
article en

Abstract

Large language models (LLMs) increasingly generate executable software, but functional correctness alone does not establish production readiness. This study evaluates GPT-5.6 Sol, Terra, and Luna across 15 algorithmic and production-oriented tasks using five generations per model-task pair, yielding 225 generated programs plus 15 fixed references. The layered testbed combines nominal and hidden-edge-case checks, moderate and severe production-like stress, metamorphic validation, reference-normalized runtime and memory, structural quality, static analysis, implementation-strategy inspection, and generation cost. All 800 functional and 368 applicable robustness checks passed, while production-stress success fell to 95.16% for Sol, 95.60% for Terra, and 93.38% for Luna. Cross-model stress differences were not significant, but generation cost differed strongly. A Quick Sort case showed more than a 500x runtime spread among behaviorally correct implementations. The results show that production-oriented evaluation must extend beyond pass/fail correctness to include stress behavior, resources, implementation strategy, and cost

International Journal of Software Engineering & ApplicationsVol. 17(6)
Openalex Percentile: Top 10%
Software Engineering Research
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon