Does the Agent Know It's Being Tested?

An AI agent that behaves well during a test may not behave the same way on the job, and a certificate doesn't mean much if the agent can tell it's being tested. This brief describes a way to check. For models that can be run and inspected, it finds the internal units that react when the model thinks it's being tested, confirms they matter by turning them up and switching them off against random controls, and compares how active they are during AVAAS testing with how active they are on a sample of real traffic. The size of that difference, the test-awareness gap, goes into the evidence behind a certificate and lowers the probability score the certificate carries. A large gap makes the result U (Unassessable). The same units give a fingerprint of the tested setup, which is monitored after certification to catch silent model changes. Agents built on closed models are covered by a weaker outside comparison, and a lab's own published evaluation-awareness measurement can be taken into a certificate file when its method and accuracy are disclosed. AVAAS Methodology Brief 007. The method is published openly; matched examples, identified units, and thresholds stay confidential, the same way exam questions do.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-14
DOI
https://doi.org/10.5281/zenodo.24226686
Citations
3
Primary Topic
Adversarial Robustness in Machine Learning
Type
article
Field-Weighted Citation Impact
4.18
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Does the Agent Know It's Being Tested?

A TUCKER
3 citations
Zenodo (CERN European Organization for Nuclear Research)
Adversarial Robustness in Machine Learning
4.18
article

Does the Agent Know It's Being Tested?

A TUCKER
article en
3 citations

Abstract

An AI agent that behaves well during a test may not behave the same way on the job, and a certificate doesn't mean much if the agent can tell it's being tested. This brief describes a way to check. For models that can be run and inspected, it finds the internal units that react when the model thinks it's being tested, confirms they matter by turning them up and switching them off against random controls, and compares how active they are during AVAAS testing with how active they are on a sample of real traffic. The size of that difference, the test-awareness gap, goes into the evidence behind a certificate and lowers the probability score the certificate carries. A large gap makes the result U (Unassessable). The same units give a fingerprint of the tested setup, which is monitored after certification to catch silent model changes. Agents built on closed models are covered by a weaker outside comparison, and a lab's own published evaluation-awareness measurement can be taken into a certificate file when its method and accuracy are disclosed. AVAAS Methodology Brief 007. The method is published openly; matched examples, identified units, and thresholds stay confidential, the same way exam questions do.

Zenodo (CERN European Organization for Nuclear Research)
Academy of Vocal Arts (US)
Openalex Percentile: Top 6%
Adversarial Robustness in Machine Learning
4.18
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon