MATER: <u>M</u> utually <u>A</u> ware Framework for <u>T</u> eleoperated-robot with <u>E</u> xtended <u>R</u> eality

Robot teleoperation with extended reality (XR teleoperation) enables intuitive interaction by mapping user motions to remote robots with real-time 3D feedback. However, existing systems suffer from large completion delays and trajectory deviations under prolonged network latency, rooted in their exclusive reliance on network communication and strict synchronous execution architecture. Moreover, network fluctuations destabilize teleoperation accuracy, while dynamic user motions amplify teleoperation errors. We present MATER, an end-to-end XR teleoperation framework that introduces a mutually-aware architecture in which each side reconstructs its counterpart's delayed or missing state to decouple the execution from network dependency. MATER includes latency-adaptive input window and user motion gap interpolation techniques to handle unstable network communication. It also proposes motion-driven robot state rollback and robot trajectory coordination to handle complex motions. The key idea behind these techniques is to adapt on the fly by reshaping reconstruction and filling gaps as network conditions fluctuate, and by realigning states when motions become fast or complex. Together with lightweight local synchronization and bandwidth optimizations, these system-level advances make MATER resilient to both network and motion dynamics. We implement MATER across three hardware settings, including simulated and physical robots, and evaluate it on 9,500 real-world teleoperation trials from the RoboSet dataset [1], covering single- and multi-step missions. Compared to state-of-the-art XR teleoperation frameworks, MATER reduces teleoperation error by up to 69.8% on WLAN and 73.1% on cellular networks with only 6.7% maximum runtime overhead. It also shortens mission completion time by up to 47.7%, enabling smoother teleoperation. A real-world case study on ten stationary and mobile missions further shows MATER achieves up to 37.7% faster completion while lowering average teleoperation error by up to 57.2%. MATER code is available at: https://github.com/rtenlab/mater