6D Pose Tracking for Adaptive AR-Mediated Human-Robot Collaboration

This paper presents a system framework for adaptive, augmented reality (AR)-mediated human-robot collaboration (HRC), enabling real-time multimodal interaction and adaptive robot behaviors during collaborative manipulation tasks. The system integrates egocentric 6D object pose tracking with real-time visual attention tracking (gaze), hand gestures, and speech, facilitating seamless two-way communication between the human and the robot. While leveraging an existing 6D pose tracker (FoundationPose), we present the first evaluation of its integration within a real-time, egocentric AR + robot framework for dynamic HRC. Our results highlight practical limitations (e.g., tracking drift) and provide design insights for building more robust, adaptive collaboration systems. The system was validated across four real-world scenarios, demonstrating promising performance and identifying key challenges for future research.