MASA: Multi Agent Situational Awareness for Visually Impaired

Visually impaired people (VIP) face difficulties in perceiving and comprehending their surroundings and, there-fore, feel disconnected from the environment. Due to visionary challenges, VIPs face challenges in carrying out routine activities but are also prone to possible risks and hazards in dynamic environments. A number of recent studies have proposed solutions for VIP based on different types of sensors and machine learning algorithms. But previously proposed systems lack an all-in-one wearable solution for visually impaired people due to rigorous training requirements for each scenario, and no conditional outputs based on user queries. In this research, we propose a novel solution named MASA for VIPs. MASA is a wearable solution for situational awareness and dynamic decision-making based on visual sensor and user queries. The proposed framework leverages vision language models (VLMs) and employs an agentic workflow for VLMs to reduce user cognitive overloading by removing all redundant information from the final output. The solution is deployed on a customized VR headset, which has Raspberry Pi, batteries, camera fitted into it. For evaluation purposes, 20 healthy subjects with good vision used MASA in 10 different scenarios. Qualitative feedback from the 20 subjects yielded encouraging results. Quantitative analysis was conducted using human-defined baselines.