Evidence-Gated Prompt Optimization: Letting Agent Systems Improve Themselves Without Trusting Them

Prompt optimizers climb whatever metric they are given, including circular ones the optimized model itself produces. We present auto-opt, a prompt optimization engine designed around distrust of its own improvements, and an evaluation of it conducted under the same posture. The engine enforces three core mechanisms in code:1. Metric-Tier Rule: Programmatic checkers (execution oracles) are always permitted; LLM judges must come from a nominally different model family verified by name lineage; declared self-labeling is refused outright.2. Held-Out Acceptance Gate: Datasets are constructed or injected and split prior to optimization. A candidate prompt is accepted and returned only if it beats the baseline by a margin on instances the optimizer never saw.3. Mandatory Risk Reporting: Every result enumerates unmeasured risks, backed by a per-rollout journal and a sealed manifest detailing models, split fingerprints, oracle provenance, and cache states. We evaluate the system across three phases: an exploratory campaign of 14 runs on small local models, three sequential adversarial audits by rival-vendor models (Grok 4.5, GPT-5.6 Terra, Gemini 2.5 Flash), and a preregistered confirmatory wave. Results demonstrate that self-judging optimizers can certify a +0.50 held-out improvement on a task where an independent oracle scores the improvement as exactly zero (a pure Goodhart gap). Conversely, enforced metric independence and held-out gating consistently filter spurious optimizations while yielding reviewable prompt diffs on valid tasks. This upload contains the preprint PDF draft, evaluation results, audit logs, and linkage to the accompanying open-source execution engine and artifact repository.