Do a Coding Model's Activations Warn of Test-Gaming Earlier than Its Text? A Pre-registered Test

When a coding agent is about to game its tests, does reading its internal activations tell you before reading its text does? I test this on HackTrace, a dataset of 173k Qwen3-8B coding attempts labelled as cheating or honest, with a pre-registered design: detectors, thresholds and decision rules were fixed before any probe was scored, the main claim was tested on a development split, and then confirmed once on a reserved set of problems that had never been opened. The answer is no. A linear probe reproduces HackTrace's in-distribution result (within-problem AUROC 0.997), but in a token-by-token replay of complete attempts at one false alarm per 20 honest attempts, it warns before the first shortcut token on 10.4% of cheats on the development split and 3.4% on the reserved set, against 11.0% and 5.6% for the best text monitor. Stronger readouts, smoothing, a monitor that reads the code written so far, and a comparison at matched false-alarm rates do not change this; a frontier judge reading each finished turn catches 92-95% of cheats before submission, more than any probe. The probe's robustness to disguised cheats (0.998 while TF-IDF falls to 0.86) is shared by a frontier judge (0.997) and by a probe on a different model's activations (0.990), so it is not specific to the model that wrote the code. On a second model family, Gemma-4-12B-it, the probe is below text monitors (0.76 vs 0.82; as a monitor, 23% vs 79% of cheats caught). Open models up to 27B almost never cheat unprompted on ImpossibleBench. For this kind of shortcut, the writing model's activations carried no early warning beyond its text that I could measure in time to act on.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-16
DOI
https://doi.org/10.5281/zenodo.24200117
Primary Topic
Academic integrity and plagiarism
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Do a Coding Model's Activations Warn of Test-Gaming Earlier than Its Text? A Pre-registered Test

Yashaswin Sharma Puranik
2 links
Zenodo (CERN European Organization for Nuclear Research)
Academic integrity and plagiarism
preprint

Do a Coding Model's Activations Warn of Test-Gaming Earlier than Its Text? A Pre-registered Test

Yashaswin Sharma Puranik
preprint en

Abstract

When a coding agent is about to game its tests, does reading its internal activations tell you before reading its text does? I test this on HackTrace, a dataset of 173k Qwen3-8B coding attempts labelled as cheating or honest, with a pre-registered design: detectors, thresholds and decision rules were fixed before any probe was scored, the main claim was tested on a development split, and then confirmed once on a reserved set of problems that had never been opened. The answer is no. A linear probe reproduces HackTrace's in-distribution result (within-problem AUROC 0.997), but in a token-by-token replay of complete attempts at one false alarm per 20 honest attempts, it warns before the first shortcut token on 10.4% of cheats on the development split and 3.4% on the reserved set, against 11.0% and 5.6% for the best text monitor. Stronger readouts, smoothing, a monitor that reads the code written so far, and a comparison at matched false-alarm rates do not change this; a frontier judge reading each finished turn catches 92-95% of cheats before submission, more than any probe. The probe's robustness to disguised cheats (0.998 while TF-IDF falls to 0.86) is shared by a frontier judge (0.997) and by a probe on a different model's activations (0.990), so it is not specific to the model that wrote the code. On a second model family, Gemma-4-12B-it, the probe is below text monitors (0.76 vs 0.82; as a monitor, 23% vs 79% of cheats caught). Open models up to 27B almost never cheat unprompted on ImpossibleBench. For this kind of shortcut, the writing model's activations carried no early warning beyond its text that I could measure in time to act on.

Zenodo (CERN European Organization for Nuclear Research)
Academic integrity and plagiarism
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon