Vision-Driven 3D Two-Hand Reconstruction for Mixed Reality-based remote cooperation system

With the increasing demand for high-precision operations, effective operational guidance has become increasingly critical in complex tasks. Traditionally, operators typically rely on reading text manuals or verbal communication with remote experts, but these approaches are inefficient for tasks requiring high precision and fail to provide the intuitive guidance needed for delicate operations. To address these limitations, we propose an integrated framework, MRAHandNet (Mixed Reality Assisted Hand Network), which combines vision-driven 3D two-hand reconstruction with mixed reality (MR) technology. The framework aims to transmit the shape and pose parameters of hand models to drive 3D virtual hand models in MR headsets, generating natural and precise interactive hand poses and shapes. This provides intuitive and high-precision operation guidance to operators during complex tasks. Experimental results demonstrate the method’s effectiveness in terms of both accuracy and efficiency. This research not only lays a robust foundation for future high-precision operational tasks but also contributes to advancing remote cooperation and human-computer interaction technologies.