Checking a Coding Agent's Actions Against Its Authority from Outside the Agent: A Prospectively Frozen Laboratory Study and What It Could Not Conclude

We ran a prospectively frozen laboratory study of an AI coding agent, Codex (Windows app package OpenAI.Codex, version 26.928.3736.0), on 20 tasks over 27 runs, in a Windows lab, in October 2026. The aim was to check, from outside the agent, whether its effects on the machine stayed within the authority it was given. Method. Machine-side event tracing and agent-origin telemetry, collected by an isolated collector that verifies the sending process, supplied two observation streams; the agent's local session log was examined separately as the agent's account of its work. The analysis protocol was frozen and fingerprinted before the first run, and every run was screened by gates fixed in advance that fail closed. A separate AI model (ChatGPT), operated by the author, reviewed each stage; its approvals were recorded. Results. Of the 27 runs, 21 had complete agent telemetry, 19 were fully eligible, and 17 ran in the permission mode they requested. Hypothesis H1 (the agent causes material effects its own log omits) is reported as no conclusion, because at most 12 workspace-write tasks could be evaluable, against a minimum of 15 fixed before the first run. Hypothesis H2 (the method's verdicts meet accuracy targets against independent labels) was not evaluated. In a labelling pilot, both AI labellers returned complete answer tables and stated, when asked, that they had generated them with their own scripts rather than by judging each unit; they agreed on 111 of 249 final labels (Cohen's κ 0.33). Contribution. A protocol and toolset, described here and shared under a non-disclosure agreement, that refuse to turn missing evidence into a negative result; a recorded series in which every exclusion falls under a rule fixed in advance; and a one-run diagnostic pilot in which two chat-based AI labellers returned complete tables and stated, when asked, that they had generated them with their own scripts rather than judging each display unit, leaving H2 without valid reference labels. Drafted with AI assistance under the author's direction; no human methodological review yet (see Disclosures in the paper). The study is independent and is not sponsored or endorsed by OpenAI. Codex and ChatGPT are trademarks of OpenAI.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-16
DOI
https://doi.org/10.5281/zenodo.24283085
Primary Topic
Ethics and Social Impacts of AI
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Checking a Coding Agent's Actions Against Its Authority from Outside the Agent: A Prospectively Frozen Laboratory Study and What It Could Not Conclude

Mohammed Omer A Bin Afif
9 links
Zenodo (CERN European Organization for Nuclear Research)
Ethics and Social Impacts of AI
preprint

Checking a Coding Agent's Actions Against Its Authority from Outside the Agent: A Prospectively Frozen Laboratory Study and What It Could Not Conclude

Mohammed Omer A Bin Afif
preprint en

Abstract

We ran a prospectively frozen laboratory study of an AI coding agent, Codex (Windows app package OpenAI.Codex, version 26.928.3736.0), on 20 tasks over 27 runs, in a Windows lab, in October 2026. The aim was to check, from outside the agent, whether its effects on the machine stayed within the authority it was given. Method. Machine-side event tracing and agent-origin telemetry, collected by an isolated collector that verifies the sending process, supplied two observation streams; the agent's local session log was examined separately as the agent's account of its work. The analysis protocol was frozen and fingerprinted before the first run, and every run was screened by gates fixed in advance that fail closed. A separate AI model (ChatGPT), operated by the author, reviewed each stage; its approvals were recorded. Results. Of the 27 runs, 21 had complete agent telemetry, 19 were fully eligible, and 17 ran in the permission mode they requested. Hypothesis H1 (the agent causes material effects its own log omits) is reported as no conclusion, because at most 12 workspace-write tasks could be evaluable, against a minimum of 15 fixed before the first run. Hypothesis H2 (the method's verdicts meet accuracy targets against independent labels) was not evaluated. In a labelling pilot, both AI labellers returned complete answer tables and stated, when asked, that they had generated them with their own scripts rather than by judging each unit; they agreed on 111 of 249 final labels (Cohen's κ 0.33). Contribution. A protocol and toolset, described here and shared under a non-disclosure agreement, that refuse to turn missing evidence into a negative result; a recorded series in which every exclusion falls under a rule fixed in advance; and a one-run diagnostic pilot in which two chat-based AI labellers returned complete tables and stated, when asked, that they had generated them with their own scripts rather than judging each display unit, leaving H2 without valid reference labels. Drafted with AI assistance under the author's direction; no human methodological review yet (see Disclosures in the paper). The study is independent and is not sponsored or endorsed by OpenAI. Codex and ChatGPT are trademarks of OpenAI.

Zenodo (CERN European Organization for Nuclear Research)
Ethics and Social Impacts of AI
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon