HEIR: Harness Egocentric Intent for Human-Robot Interactions
For robots to assist, serve, and collaborate with humans, they must not only interpret human requests conveyed through language and multimodal intentional cues, but also execute long-horizon tasks that interleave navigation and manipulation. In this paper, we introduce HEIR, a resource for studying how to Harness Egocentric Intent for Human-Robot Interaction. HEIR pairs human speech, measured gaze, and egocentric video with robot observations and teleoperated navigation-manipulation trajectories, while each session preserves successive episodes, with each request fulfilled through multiple execution subtasks. The captured data, together with human-authored annotations, support the study of intent grounding, execution planning, and task completion within a unified interaction setting. To demonstrate the value of HEIR, we conduct extensive offline experiments to evaluate how models benefit from additional human-centric context across these tasks, together with real-robot experiments that assess execution task completion. HEIR thus provides a valuable resource for studying user-centric robot interaction in naturalistic household environments. Our demo and dataset are available online.
Authors
- 賴宗炫
- Guocai Yao (ORCID: https://orcid.org/0000-0002-0673-0229)
- Shaoxuan Xie
- Haoyang Shi (ORCID: https://orcid.org/0000-0002-6917-5007)
- Kejun Gao
- Sen Cui
- Chi Zhang
- Yutong Han
- Chang Liu
- Bo Wang
- Miao Liu
- Zhenghan Zhu
- Yichi Zhang
- Jiayin Zou
Publication Details
- Journal
- arXiv (Cornell University)
- Published
- 2026-11-12
- DOI
- https://doi.org/10.48550/arxiv.2611.17238
- Primary Topic
- Social Robot Interaction and HRI
- Type
- preprint