Enabling Real-World Assistive Agents: From Live Vision to Proactive Context-Aware Information Delivery

Figure 1: My dissertation work explores the design of an assistive agent as an always-available companion to assist BVI people in real-world interactions.(a) I first conducted a user study with eight BVI people to probe the design insights for designing assistive agents [11].Based on the insights, (b) I developed a system, WorldScribe, that proactively provides live visual descriptions based on user intent, and visual and sound contexts [10].(c) My ongoing works expand WorldScribe to take natural user interactions to provide corresponding information for BVI people, such as object descriptions for hand-object interactions, and spatial information based on location and mobility.(d) In the future, I aim to draw insights from human assistance to develop an assistive agent that serves as a long-term, context-aware companion in everyday environments.