How GUI Agents Receive Instructions: An Exploratory Mapping of 85 Benchmark Families
Preprint. An exploratory mapping of how 85 GUI and computer-use agent benchmark families give the agent its task, coded for a declared, non-exhaustive frame of 101 candidates (a keyword-targeted stratum and a seeded random comparison sample). The paper separates three constructs that descriptions of benchmarks often mix: how the goal is specified, what the agent observes while acting, and how completion is judged. In both strata the goal is given as natural-language text in almost every component; a video, recording or demonstration is part of the specification in 9 of 81 components of the targeted stratum and 1 of 66 of the comparison stratum, and more components are coded as using video as authoring input (18 of 81 in the targeted stratum, some thinly evidenced) than as a specification channel. All coding was done by AI agents and audited by a second AI pass; no independent human recoding was performed, and the study was not preregistered. Counts describe the sample, not the field. Competing interest: The author is affiliated with Vertov (https://vertov.app), a company developing screen-recording and AI-agent collaboration software. Vertov has a commercial interest in the subject of this research. Companion dataset: see related identifiers.
Authors
- Ankur Shrestha (ORCID: https://orcid.org/0009-0009-6036-5849)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-14
- DOI
- https://doi.org/10.5281/zenodo.24217328
- Primary Topic
- Usability and User Interface Design
- Type
- preprint