Failures Are Rocket Fuel: Verified Per-Model Memory for Local Models in the Unity Editor

A recent census of 10,400 single-pass Unity generations from four open-weight 7-30B models found that none compiled into a runnable scene, and concluded that the bottleneck is missing engine-specific knowledge. We agree with the diagnosis and disagree that it is a ceiling. With the editor kept in the loop and the missing knowledge supplied as context at run time, three local 7B-14B models complete 95.0-98.3% of 48 Unity 2022.3 editor tasks (10 runs each, every verdict read from the real scene in a headless editor). We then give each model a verified record of its own: only outcomes a verifier judged, with what the model claimed beside what the editor found, recalled before each request and written back only as verified. For live work without a specification, reading the user's request with the same reader that checks the model's claims raises the precision of a "win" from 96.35% to 99.64% on 10,903 recorded completions, losing no real win. With its record, a model passes more of its weakest tasks: pooled over three models and seven tasks, 139 of 210 runs without a record, 181 with its own record and 179 with a record that never held those tasks (CMH p = 2e-07 and 3e-07). The gain is two models' (Morpheus 14B: 40 to 60 and 59 of 60; Gemma 4 12B: 60 to 82 and 89 of 90); for Qwen 7B the record did not help. The transcripts show why: models copy the form of the record's lines. Two cautions came from the same measurements: about 8,000 tokens of off-task chat history cost every model (Morpheus 14B: 77 to 59 of 80, p = 2e-05), and an ungated memory teaches false lessons (1,211 of one model's recorded failures were the harness's fault). Built with Claude (Anthropic) in Claude Code; see the paper's acknowledgements.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-13
DOI
https://doi.org/10.5281/zenodo.24183725
Primary Topic
Software Engineering Research
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Failures Are Rocket Fuel: Verified Per-Model Memory for Local Models in the Unity Editor

Michael Ricky Neal
3 links
Zenodo (CERN European Organization for Nuclear Research)
Software Engineering Research
preprint

Failures Are Rocket Fuel: Verified Per-Model Memory for Local Models in the Unity Editor

Michael Ricky Neal
preprint en

Abstract

A recent census of 10,400 single-pass Unity generations from four open-weight 7-30B models found that none compiled into a runnable scene, and concluded that the bottleneck is missing engine-specific knowledge. We agree with the diagnosis and disagree that it is a ceiling. With the editor kept in the loop and the missing knowledge supplied as context at run time, three local 7B-14B models complete 95.0-98.3% of 48 Unity 2022.3 editor tasks (10 runs each, every verdict read from the real scene in a headless editor). We then give each model a verified record of its own: only outcomes a verifier judged, with what the model claimed beside what the editor found, recalled before each request and written back only as verified. For live work without a specification, reading the user's request with the same reader that checks the model's claims raises the precision of a "win" from 96.35% to 99.64% on 10,903 recorded completions, losing no real win. With its record, a model passes more of its weakest tasks: pooled over three models and seven tasks, 139 of 210 runs without a record, 181 with its own record and 179 with a record that never held those tasks (CMH p = 2e-07 and 3e-07). The gain is two models' (Morpheus 14B: 40 to 60 and 59 of 60; Gemma 4 12B: 60 to 82 and 89 of 90); for Qwen 7B the record did not help. The transcripts show why: models copy the form of the record's lines. Two cautions came from the same measurements: about 8,000 tokens of off-task chat history cost every model (Morpheus 14B: 77 to 59 of 80, p = 2e-05), and an ungated memory teaches false lessons (1,211 of one model's recorded failures were the harness's fault). Built with Claude (Anthropic) in Claude Code; see the paper's acknowledgements.

Zenodo (CERN European Organization for Nuclear Research)
Software Engineering Research
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon