When Free Executors Cost More: The Free-Executor Paradox in Iterative LLM Code-Repair Loops

We compare four LLM configurations on three Python code-repair tasks (breakage recovery, refactor, feature-add) using a single deterministic correctness judge (mypy + ruff + pytest exit codes). The configurations are two single-model 'solo' runs of Anthropic Opus 4.7 and Haiku 4.5, and two 'orchestrator + executor' structures in which Opus orchestrates either local Qwen 3.5-9B (zero marginal cost) or Anthropic Haiku 4.5. Across 40 trials with n=3 successful runs per cell, the cost/wall-time/iterations Pareto frontier varies across the three task types. On the smallest task, Opus solo dominates every cloud-only axis at $0.17/trial and 6 iterations median; only Haiku solo undercuts it on cost. On the longest task, Haiku solo wins on dollar cost (5.5x cheaper than the cheapest cloud arm), at a 25% failure rate within our per-arm iteration cap. The canonical 'strong orchestrator + cheap executor' structure (Opus + Qwen) is the most expensive cloud arm on every task, because the orchestrator's prompt-cached re-reads of the executor's returned summaries dominate any savings from delegating execution. The paper aims to make the underlying tradeoff cheap to measure on other codebases, not to crown a single winner.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-06-28
DOI
https://doi.org/10.5281/zenodo.20978073
Primary Topic
Software Engineering Research
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

When Free Executors Cost More: The Free-Executor Paradox in Iterative LLM Code-Repair Loops

Ken Imoto
Zenodo (CERN European Organization for Nuclear Research)
Software Engineering Research
preprint

When Free Executors Cost More: The Free-Executor Paradox in Iterative LLM Code-Repair Loops

Ken Imoto
preprint en

Abstract

We compare four LLM configurations on three Python code-repair tasks (breakage recovery, refactor, feature-add) using a single deterministic correctness judge (mypy + ruff + pytest exit codes). The configurations are two single-model 'solo' runs of Anthropic Opus 4.7 and Haiku 4.5, and two 'orchestrator + executor' structures in which Opus orchestrates either local Qwen 3.5-9B (zero marginal cost) or Anthropic Haiku 4.5. Across 40 trials with n=3 successful runs per cell, the cost/wall-time/iterations Pareto frontier varies across the three task types. On the smallest task, Opus solo dominates every cloud-only axis at $0.17/trial and 6 iterations median; only Haiku solo undercuts it on cost. On the longest task, Haiku solo wins on dollar cost (5.5x cheaper than the cheapest cloud arm), at a 25% failure rate within our per-arm iteration cap. The canonical 'strong orchestrator + cheap executor' structure (Opus + Qwen) is the most expensive cloud arm on every task, because the orchestrator's prompt-cached re-reads of the executor's returned summaries dominate any savings from delegating execution. The paper aims to make the underlying tradeoff cheap to measure on other codebases, not to crown a single winner.

Zenodo (CERN European Organization for Nuclear Research)
Electric Propulsion Laboratory (United States) (US)
Software Engineering Research
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

When Free Executors Cost More: The Free-Executor Paradox in Iterative LLM Code-Repair Loops — Ken Imoto · Zenodo (CERN European Organization for Nuclear Research) (2026) | TGRS Research Map | TGRS