Assistive AR System for Enhancing Human-Human and Human-Environment Interactions

Augmented Reality (AR) smart glasses promise transformative enhancements to human interactions and environmental awareness, yet face substantial challenges, including limited computational resources, constrained sensing capabilities, and inadequate intuitive interfaces. We introduce Assistive AR, a new category of AR systems designed explicitly for assistance to people's everyday lives, supporting both human-human and human-environment interactions through human-like sensing, efficient computation, and intuitive presentations. Specifically, Sign-to-911 addresses communication barriers for deaf users during emergencies by providing real-time bidirectional Sign Language translation on the AR system. SocialMind enhances social interactions using proactive large language models (LLMs) to interpret and suggest responses based on conversational context. Further, Sensor2Scene and Vivar utilize generative models to visualize complex environmental sensor data to enhance the human-environment interaction. Moving forward, we will provide a general and scalable framework called MASG-MCP that enables users to create and customize tools on the Assistive AR system through a unified LLM agent.