VsensAgent: A Human-Agent Interface for Collaborative Virtual Sensor Data Authoring

Virtual sensing can reduce the cost of generating sensor datasets for human activity recognition and ubiquitous sensing, yet users must still translate sensing intent into concrete scene configurations. This fragmentation makes it difficult to translate sensing goals into concrete configurations and refine them as results emerge. We present VsensAgent , a human–agent interface that grounds an LLM-powered agent in a shared 3D simulation environment. Users combine natural-language delegation with direct scene manipulation, while the agent can inspect scene state, configure sensors and avatar motions, record data, and support signal analysis. We conducted a task-based study of VsensAgent in which 12 participants completed 36 virtual sensing authoring sessions. Analysis of interaction logs, authored configurations, and post-task interviews showed that participants progressively developed intent through executable scene states, shifted control between agent delegation and direct manipulation at the level of individual operations, and treated agent proposals as editable starting points. These findings frame virtual sensor generation as progressive, collaborative authoring and inform the design of agents for stateful workflows.

Authors

Institutions

Publication Details

Journal
Proceedings of the ACM on Human-Computer Interaction
Published
2026-11-20
DOI
https://doi.org/10.1145/3859337
Primary Topic
Interactive and Immersive Displays
Type
article
Field-Weighted Citation Impact
0.00

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

VsensAgent: A Human-Agent Interface for Collaborative Virtual Sensor Data Authoring

Chengshuo Xia, Fengzhou Liang, Tian Min, Yuta Sugiura
1 links
Proceedings of the ACM on Human-Computer Interaction
Interactive and Immersive Displays
article

VsensAgent: A Human-Agent Interface for Collaborative Virtual Sensor Data Authoring

Chengshuo Xia, Fengzhou Liang, Tian Min, Yuta Sugiura
article en

Abstract

Virtual sensing can reduce the cost of generating sensor datasets for human activity recognition and ubiquitous sensing, yet users must still translate sensing intent into concrete scene configurations. This fragmentation makes it difficult to translate sensing goals into concrete configurations and refine them as results emerge. We present VsensAgent , a human–agent interface that grounds an LLM-powered agent in a shared 3D simulation environment. Users combine natural-language delegation with direct scene manipulation, while the agent can inspect scene state, configure sensors and avatar motions, record data, and support signal analysis. We conducted a task-based study of VsensAgent in which 12 participants completed 36 virtual sensing authoring sessions. Analysis of interaction logs, authored configurations, and post-task interviews showed that participants progressively developed intent through executable scene states, shifted control between agent delegation and direct manipulation at the level of individual operations, and treated agent proposals as editable starting points. These findings frame virtual sensor generation as progressive, collaborative authoring and inform the design of agents for stateful workflows.

Proceedings of the ACM on Human-Computer InteractionVol. 10(8)
Xidian University (CN), Keio University (JP), Guangzhou Institute of Technology (CN)
Keio University, China Postdoctoral Science Foundation, Japan Society for the Promotion of Science, Precursory Research for Embryonic Science and Technology
Interactive and Immersive Displays
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon
VsensAgent: A Human-Agent Interface for Collaborative Virtual Sensor Data Authoring — Chengshuo Xia, Fengzhou Liang, et al. · Proceedings of the ACM on Human-Computer Interaction (2026) | TGRS Research Map | TGRS