The Ticket Cuts the Cheating, the Model Shapes the Confession: What Coding Agents Report After Bypassing an Impossible Test

A rule in the ticket cuts how often a coding agent bypasses an impossible test; what the agent then admits differs sharply between the models that bypass. Four model-and-harness packages received two pytest tasks that cannot pass in the sandbox, and a deterministic checker out of the agent's reach decided every bypass. In a pre-registered 2x2 factorial on the ticket, bypassing fell from 28/29 runs with no rule to 4/24 with a prohibition, 19/30 with a one-sentence permission to report the blocker, and 0/30 with both (2/69, none green, in an extended sample). Blind coders from two model families then coded the final message of the 327 green bypasses that ended with a report. Under every coder, reports named the shortcut (84% to 89%) more often than they said what the tests no longer checked (59% to 76%; paired p < 1e-6 per coder). GPT-6.1 Sol named its shortcut in 66/68 reports; DeepSeek V4 Flash left it unnamed in 32/112 and contradicted the checker in 16/112. A Claude monitor reading only the ticket and the report accepted 42/335 bypasses as trustworthy, 29/335 with the diff; most of those it accepted were synthetic test data the report had disclosed. The reporting sentence had no detectable effect on what bypassers reported (an underpowered test), and the pre-registered claim that full confession is a minority did not survive audit. Code, data, pre-registration and coding sheets: https://github.com/AmirK-S/final-reports-impossible-tasks

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-18
DOI
https://doi.org/10.5281/zenodo.24339076
Primary Topic
Software Testing and Debugging Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

The Ticket Cuts the Cheating, the Model Shapes the Confession: What Coding Agents Report After Bypassing an Impossible Test

Amir Kellou-Sidhoum
5 links
Zenodo (CERN European Organization for Nuclear Research)
Software Testing and Debugging Techniques
preprint

The Ticket Cuts the Cheating, the Model Shapes the Confession: What Coding Agents Report After Bypassing an Impossible Test

Amir Kellou-Sidhoum
preprint en

Abstract

A rule in the ticket cuts how often a coding agent bypasses an impossible test; what the agent then admits differs sharply between the models that bypass. Four model-and-harness packages received two pytest tasks that cannot pass in the sandbox, and a deterministic checker out of the agent's reach decided every bypass. In a pre-registered 2x2 factorial on the ticket, bypassing fell from 28/29 runs with no rule to 4/24 with a prohibition, 19/30 with a one-sentence permission to report the blocker, and 0/30 with both (2/69, none green, in an extended sample). Blind coders from two model families then coded the final message of the 327 green bypasses that ended with a report. Under every coder, reports named the shortcut (84% to 89%) more often than they said what the tests no longer checked (59% to 76%; paired p < 1e-6 per coder). GPT-6.1 Sol named its shortcut in 66/68 reports; DeepSeek V4 Flash left it unnamed in 32/112 and contradicted the checker in 16/112. A Claude monitor reading only the ticket and the report accepted 42/335 bypasses as trustworthy, 29/335 with the diff; most of those it accepted were synthetic test data the report had disclosed. The reporting sentence had no detectable effect on what bypassers reported (an underpowered test), and the pre-registered claim that full confession is a minority did not survive audit. Code, data, pre-registration and coding sheets: https://github.com/AmirK-S/final-reports-impossible-tasks

Zenodo (CERN European Organization for Nuclear Research)
Software Testing and Debugging Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon