How GUI Agents Receive Instructions: An Exploratory Mapping of 85 Benchmark Families

Preprint. An exploratory mapping of how 85 GUI and computer-use agent benchmark families give the agent its task, coded for a declared, non-exhaustive frame of 101 candidates (a keyword-targeted stratum and a seeded random comparison sample). The paper separates three constructs that descriptions of benchmarks often mix: how the goal is specified, what the agent observes while acting, and how completion is judged. In both strata the goal is given as natural-language text in almost every component; a video, recording or demonstration is part of the specification in 9 of 81 components of the targeted stratum and 1 of 66 of the comparison stratum, and more components are coded as using video as authoring input (18 of 81 in the targeted stratum, some thinly evidenced) than as a specification channel. All coding was done by AI agents and audited by a second AI pass; no independent human recoding was performed, and the study was not preregistered. Counts describe the sample, not the field. Competing interest: The author is affiliated with Vertov (https://vertov.app), a company developing screen-recording and AI-agent collaboration software. Vertov has a commercial interest in the subject of this research. Companion dataset: see related identifiers.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-14
DOI
https://doi.org/10.5281/zenodo.24217328
Primary Topic
Usability and User Interface Design
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

How GUI Agents Receive Instructions: An Exploratory Mapping of 85 Benchmark Families

Ankur Shrestha
Zenodo (CERN European Organization for Nuclear Research)
Usability and User Interface Design
preprint

How GUI Agents Receive Instructions: An Exploratory Mapping of 85 Benchmark Families

Ankur Shrestha
preprint en

Abstract

Preprint. An exploratory mapping of how 85 GUI and computer-use agent benchmark families give the agent its task, coded for a declared, non-exhaustive frame of 101 candidates (a keyword-targeted stratum and a seeded random comparison sample). The paper separates three constructs that descriptions of benchmarks often mix: how the goal is specified, what the agent observes while acting, and how completion is judged. In both strata the goal is given as natural-language text in almost every component; a video, recording or demonstration is part of the specification in 9 of 81 components of the targeted stratum and 1 of 66 of the comparison stratum, and more components are coded as using video as authoring input (18 of 81 in the targeted stratum, some thinly evidenced) than as a specification channel. All coding was done by AI agents and audited by a second AI pass; no independent human recoding was performed, and the study was not preregistered. Counts describe the sample, not the field. Competing interest: The author is affiliated with Vertov (https://vertov.app), a company developing screen-recording and AI-agent collaboration software. Vertov has a commercial interest in the subject of this research. Companion dataset: see related identifiers.

Zenodo (CERN European Organization for Nuclear Research)
Usability and User Interface Design
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon