Empirical Evaluation of Iterative Error-Feedback Loops in Automated TypeScript Unit Test Generation: A Cost-Efficiency and Turn-Budget Perspective

Automated unit test generation using Large Language Models (LLMs) has emerged as a promising methodology to alleviate developer burden in continuous integration (CI) lifecycles. However, unguided single-shot generation often yields test suites plagued by subtle assertion errors and interface discrepancies. While recent literature demonstrates that iterative validation-and-repair loops improve test pass rates in dynamically typed environments like Python, empirical investigations into strictly typed environments—specifically TypeScript—remain scarce. In this empirical study, we investigate the multi-turn error-feedback loop for unit test generation across 30 diverse TypeScript functions categorized into three complexity tiers (algorithmic utilities, data validation, and asynchronous logic). We evaluate four cost-effective frontier models: Google Gemini 3.8 Flash, DeepSeek-V4.1 Flash, Alibaba Qwen 3.8 Flash, and Zhipu AI GLM-5.3 Flash, with an execution repair budget of up to K=3 iterations. We formulate four research questions addressing: (1) pass rate and repair convergence dynamics across turns and complexity tiers, (2) turn-budget saturation and error stagnation, (3) comparative model performance and Cost-per-Passing-Test (CPT), and (4) a failure taxonomy distinguishing compilation errors from runtime assertion failures. Our empirical findings reveal that three of the four models (Gemini 3.8 Flash, Qwen 3.8 Flash, and GLM-5.3 Flash) achieve a flawless 100.0% final pass rate, while DeepSeek-V4.1 Flash attains 96.7%, yielding an aggregate benchmark pass rate of 99.2% (119/120 test suites) across all evaluated cases. Crucially, a single feedback iteration (K=1) successfully recovers 94.7% (18/19) of initial failures, whereas subsequent iterations (K=2 and K=3) yield exactly 0.0% marginal gain, establishing a strict empirical stopping rule of K* = 1 for production CI/CD pipelines. Furthermore, GLM-5.3 Flash emerges as the most cost-efficient model at $0.00051 per passing test (5.36× cheaper than Gemini 3.8 Flash). Finally, we demonstrate that Server-Sent Events (SSE) streaming is an essential architectural requirement for reasoning-capable models to eliminate gateway read timeouts. This study provides actionable, cost-calibrated guidelines for engineering teams deploying LLMs for automated software testing.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-18
DOI
https://doi.org/10.5281/zenodo.24335203
Primary Topic
Software Testing and Debugging Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Empirical Evaluation of Iterative Error-Feedback Loops in Automated TypeScript Unit Test Generation: A Cost-Efficiency and Turn-Budget Perspective

Syauqi Fuadi
3 links
Zenodo (CERN European Organization for Nuclear Research)
Software Testing and Debugging Techniques
preprint

Empirical Evaluation of Iterative Error-Feedback Loops in Automated TypeScript Unit Test Generation: A Cost-Efficiency and Turn-Budget Perspective

Syauqi Fuadi
preprint en

Abstract

Automated unit test generation using Large Language Models (LLMs) has emerged as a promising methodology to alleviate developer burden in continuous integration (CI) lifecycles. However, unguided single-shot generation often yields test suites plagued by subtle assertion errors and interface discrepancies. While recent literature demonstrates that iterative validation-and-repair loops improve test pass rates in dynamically typed environments like Python, empirical investigations into strictly typed environments—specifically TypeScript—remain scarce. In this empirical study, we investigate the multi-turn error-feedback loop for unit test generation across 30 diverse TypeScript functions categorized into three complexity tiers (algorithmic utilities, data validation, and asynchronous logic). We evaluate four cost-effective frontier models: Google Gemini 3.8 Flash, DeepSeek-V4.1 Flash, Alibaba Qwen 3.8 Flash, and Zhipu AI GLM-5.3 Flash, with an execution repair budget of up to K=3 iterations. We formulate four research questions addressing: (1) pass rate and repair convergence dynamics across turns and complexity tiers, (2) turn-budget saturation and error stagnation, (3) comparative model performance and Cost-per-Passing-Test (CPT), and (4) a failure taxonomy distinguishing compilation errors from runtime assertion failures. Our empirical findings reveal that three of the four models (Gemini 3.8 Flash, Qwen 3.8 Flash, and GLM-5.3 Flash) achieve a flawless 100.0% final pass rate, while DeepSeek-V4.1 Flash attains 96.7%, yielding an aggregate benchmark pass rate of 99.2% (119/120 test suites) across all evaluated cases. Crucially, a single feedback iteration (K=1) successfully recovers 94.7% (18/19) of initial failures, whereas subsequent iterations (K=2 and K=3) yield exactly 0.0% marginal gain, establishing a strict empirical stopping rule of K* = 1 for production CI/CD pipelines. Furthermore, GLM-5.3 Flash emerges as the most cost-efficient model at $0.00051 per passing test (5.36× cheaper than Gemini 3.8 Flash). Finally, we demonstrate that Server-Sent Events (SSE) streaming is an essential architectural requirement for reasoning-capable models to eliminate gateway read timeouts. This study provides actionable, cost-calibrated guidelines for engineering teams deploying LLMs for automated software testing.

Zenodo (CERN European Organization for Nuclear Research)
Software Testing and Debugging Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon