XRMan: Towards Real-time Hand-Object Pose Tracking in eXtended Reality

Accurate tracking of hand and object poses is essential for immersive XR applications. However, existing methods often struggle with the stringent requirements of XR environments. We present XRMan, a real-time hand-object pose tracking system designed for in-the-wild scenarios. XRMan uses a drift monitoring module for consistent accuracy, while farthest-point sampling and collider-based optimization streamline iterative optimization and reduce latency. Our system reduces end-to-end latency by 38.9% compared to the baseline, with further improvements expected from the collider-based post-optimization module.