Transforming Procedural Instructions into In-Situ Augmented Reality Guides with InstructAR

Following manual procedural instructions can often be frustrating due to the mental gap between the instructions and their real-world application. Variations in settings can further compound this confusion for those attempting to follow them. To address this problem, we propose InstructAR, a system under development that bridges textual instructions with the real world through Augmented Reality (AR) guides. With this tool, users can easily follow clear and actionable step-by-step instructions, as related objects are highlighted and AR guides indicate necessary actions. InstructAR also provides feedback upon task completion. This proof-of-concept leverages Natural Language Processing and Computer Vision techniques. Our work aims to reduce cognitive load and errors, making it easier for users to follow manual procedural instructions accurately.