Investigating User Experience with Proactive Agents Based on LLMs in Home Automation

Objective: This study aims to propose and validate a proactive agent framework that enhances usability and interaction efficiency through context-aware assistance in smart home environments. The potential of the framework is evaluated through a scenario-based user study using LLM-powered conversational agents.<br/><br/>Background: Advances in large language models (LLMs) have accelerated the development of proactive conversational agents that infer user intentions from context and provide support without explicit requests. While these systems address limitations of reactive agents, their usability and effectiveness depend on how well they align with the user's contextual needs.<br/><br/>Method: We conducted a within-subjects study with 18 participants to compare a proactive agent based on our framework with a reactive baseline. Participants completed six tasks across three smart home scenarios using a chat interface powered by LLMs. Usability and interaction efficiency were measured through surveys and prompt analysis, supported by qualitative insights from follow-up interviews.<br/><br/>Results: The proactive agent based on our framework demonstrated significantly higher usability, as measured by the SUS score. It also reduced prompt verbosity and quantity compared to the reactive agent, indicating improved interaction efficiency. Interview responses further supported these findings, with most participants preferring the proactive agent for its convenience, reduced cognitive effort, and ability to streamline routine automation through intuitive suggestions.<br/><br/>Conclusion: The results confirm the effectiveness of a user-centered, modular proactive agent framework in enhancing smart home interaction. Furthermore, the findings underscore the importance of personalized suggestion delivery, integration of user expertise, and mechanisms for transparent refinement based on user feedback.<br/><br/>Application: These findings offer design implications for more broadly applying proactive systems for automation beyond smart homes.