An efficient auxiliary system for mechanical maintenance based on 3D object tracking

The Augmented Reality for Maintenance and Repair (ARMAR) system aims to provide an intuitive operation guidance by mixing digital information with reality. As an impressive work, most auxiliary systems in headmounted display devices show the rendered digital labels, texts, or other prompts to the users after 2D detection, that is still far to apply in practice. To solve these issues namely target occlusion, rapid movement, and metallic surfaces in real-world scenarios, we develope an efficient auxiliary system based on 3D object tracking which consists of three states. First, we use simultaneous localization and mapping (SLAM) to position the entire space, the target is rendered at the correct position as a reference. Secondly, we detect the object with YOLOv8 network and match it with this object’s pose templates to estimate the initial pose of the target. Finally, we track the target with an improved CAD model based method. To verify the system, we collect metal object related data in real mechanical maintenance scenarios for the test. The results show that our system is robust to occlusion and object loss due to motion and metallic surfaces in practical applications, and it achieves real-time speed on RK3588 platform.