The Effect of Hand Visibility in AR: Comparing Dexterity and Interaction with Virtual and Real Objects

Hand-tracking technologies allow us to use our own hands to interact with real and virtual objects in Augmented Reality (AR) environments. This enables us to explore the interplay between hand-visualizations and hand-object interactions. We present a user study that examines the effect of different hand visualizations (invisible, transparent, opaque) on manipulation performance when interacting with real and virtual objects. For this, we implemented video-see-through (VST) AR-based virtual building blocks and hot wire tasks with real one-to-one counterparts that require participants to use gross and fine motor hand movements. To evaluate manipulation performance, we considered three measures: task completion time, number of collisions (hot wire task), and percentage of object displacement (building block task). Additionally, we explored the sense of agency and subjective impressions (preference, ease of interaction, successful and awkwardness) evoked by the different hand-visualizations. The results show that (1) manipulation performance is significantly higher when interacting with real objects compared to virtual ones, (2) invisible hands lead to fewer errors, higher agency, higher perceived success and ease of interaction during fine manipulation tasks with real objects, and (3) having some visualization of the virtual hands (transparent or opaque) overlayed on the real hands is preferred when manipulating virtual objects even when there are no significant performance improvements. Our empirical findings about the differences when interacting with real and virtual objects can aid hand visualization choices for manipulation tasks in AR.