Egocentric expert guidance for procedural activities using Smart Glasses, Computer Vision and RAG

The paper proposes an automated guidance system designed to replicate the role of a domain expert in re-manufacturing scenarios. Our system aims to provide non-experts with real-time advice and instructions on next steps when carrying out procedural tasks. Using smart glasses to capture an egocentric view of the environment and accessing structured domain knowledge, the system can retrieve information to guide users effectively. By breaking down complex tasks into smaller, manageable subtasks, similar to how humans naturally organize activities, the system helps users understand and complete each step. It incorporates a procedural memory that, like human memory, tracks completed subtasks and integrates with smart glasses that capture visual and auditory inputs. These inputs allow the system to recognize a given state as a specific, chronologically ordered key step. This state information is used in a retrieval-augmented generation prompt, which interacts with a knowledge graph grounded in expert knowledge. As a result, the system provides reliable, state-specific instructions and advice for each next step. Our findings highlight the potential of wearable technology to enhance human performance in industrial environments by providing expert-informed task support.