VsensAgent: A Human-Agent Interface for Collaborative Virtual Sensor Data Authoring
Virtual sensing can reduce the cost of generating sensor datasets for human activity recognition and ubiquitous sensing, yet users must still translate sensing intent into concrete scene configurations. This fragmentation makes it difficult to translate sensing goals into concrete configurations and refine them as results emerge. We present VsensAgent , a human–agent interface that grounds an LLM-powered agent in a shared 3D simulation environment. Users combine natural-language delegation with direct scene manipulation, while the agent can inspect scene state, configure sensors and avatar motions, record data, and support signal analysis. We conducted a task-based study of VsensAgent in which 12 participants completed 36 virtual sensing authoring sessions. Analysis of interaction logs, authored configurations, and post-task interviews showed that participants progressively developed intent through executable scene states, shifted control between agent delegation and direct manipulation at the level of individual operations, and treated agent proposals as editable starting points. These findings frame virtual sensor generation as progressive, collaborative authoring and inform the design of agents for stateful workflows.
Authors
- Chengshuo Xia (ORCID: https://orcid.org/0000-0002-3937-2077)
- Fengzhou Liang (ORCID: https://orcid.org/0000-0002-1459-0385)
- Tian Min (ORCID: https://orcid.org/0000-0003-0912-4598)
- Yuta Sugiura (ORCID: https://orcid.org/0000-0003-3735-4809)
Institutions
- Xidian University (CN)
- Keio University (JP)
- Guangzhou Institute of Technology (CN)
Publication Details
- Journal
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2026-11-20
- DOI
- https://doi.org/10.1145/3859337
- Primary Topic
- Interactive and Immersive Displays
- Type
- article
- Field-Weighted Citation Impact
- 0.00
Funders
- Keio University
- China Postdoctoral Science Foundation
- Japan Society for the Promotion of Science
- Precursory Research for Embryonic Science and Technology