Failures Are Rocket Fuel: Verified Per-Model Memory for Local Models in the Unity Editor
A recent census of 10,400 single-pass Unity generations from four open-weight 7-30B models found that none compiled into a runnable scene, and concluded that the bottleneck is missing engine-specific knowledge. We agree with the diagnosis and disagree that it is a ceiling. With the editor kept in the loop and the missing knowledge supplied as context at run time, three local 7B-14B models complete 95.0-98.3% of 48 Unity 2022.3 editor tasks (10 runs each, every verdict read from the real scene in a headless editor). We then give each model a verified record of its own: only outcomes a verifier judged, with what the model claimed beside what the editor found, recalled before each request and written back only as verified. For live work without a specification, reading the user's request with the same reader that checks the model's claims raises the precision of a "win" from 96.35% to 99.64% on 10,903 recorded completions, losing no real win. With its record, a model passes more of its weakest tasks: pooled over three models and seven tasks, 139 of 210 runs without a record, 181 with its own record and 179 with a record that never held those tasks (CMH p = 2e-07 and 3e-07). The gain is two models' (Morpheus 14B: 40 to 60 and 59 of 60; Gemma 4 12B: 60 to 82 and 89 of 90); for Qwen 7B the record did not help. The transcripts show why: models copy the form of the record's lines. Two cautions came from the same measurements: about 8,000 tokens of off-task chat history cost every model (Morpheus 14B: 77 to 59 of 80, p = 2e-05), and an ungated memory teaches false lessons (1,211 of one model's recorded failures were the harness's fault). Built with Claude (Anthropic) in Claude Code; see the paper's acknowledgements.
Authors
- Michael Ricky Neal
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-13
- DOI
- https://doi.org/10.5281/zenodo.24183725
- Primary Topic
- Software Engineering Research
- Type
- preprint