A “Human-Like” Stroop Effect No Longer Implies a Human Participant

Researchers have long assumed that online behavioral experiments are completed by humans, an assumption AI agents can now violate. We asked whether agent and human data can be distinguished. We compared Stroop data from 51 respondents each: humans verified in person, an artificial experiment agent, and an online sample. All agent-generated respondents showed the congruency effect and passed six quality checks. A one-class support vector machine (SVM) trained only on human data achieved a median accuracy of 63%. A supervised SVM trained on human and agent data achieved 94%. At an assumed 5% contamination rate, most of its flags would be false alarms. The supervised SVM did not generalize to an agent it was not trained on. Neither plausible task performance nor the tested SVMs reliably distinguished human from agent data. These findings suggest that human participation should be safeguarded through study design rather than through screening afterward.

Authors

Publication Details

Journal
PsyArXiv (OSF Preprints)
Published
2026-11-13
DOI
https://doi.org/10.31234/osf.io/ufpgn
Primary Topic
Mobile Crowdsensing and Crowdsourcing
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

A “Human-Like” Stroop Effect No Longer Implies a Human Participant

Leendert van Maanen, Christian P. Janssen, Christoph Strauch, Talha Ozudogru et al.
3 links
PsyArXiv (OSF Preprints)
Mobile Crowdsensing and Crowdsourcing
preprint

A “Human-Like” Stroop Effect No Longer Implies a Human Participant

Leendert van Maanen, Christian P. Janssen, Christoph Strauch, Talha Ozudogru, Silja Renooij
preprint en

Abstract

Researchers have long assumed that online behavioral experiments are completed by humans, an assumption AI agents can now violate. We asked whether agent and human data can be distinguished. We compared Stroop data from 51 respondents each: humans verified in person, an artificial experiment agent, and an online sample. All agent-generated respondents showed the congruency effect and passed six quality checks. A one-class support vector machine (SVM) trained only on human data achieved a median accuracy of 63%. A supervised SVM trained on human and agent data achieved 94%. At an assumed 5% contamination rate, most of its flags would be false alarms. The supervised SVM did not generalize to an agent it was not trained on. Neither plausible task performance nor the tested SVMs reliably distinguished human from agent data. These findings suggest that human participation should be safeguarded through study design rather than through screening afterward.

PsyArXiv (OSF Preprints)
Mobile Crowdsensing and Crowdsourcing
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon