Design of a Human-Centric Robotic System for User Support Based on Gaze Information

Recent advancements in mechanization and automation have significantly transformed households and retail environments, with automated services becoming increasingly prevalent. In general, smart appliances utilizing the IoT technology have gained widespread adoption, and computerized systems, such as self-checkout machines, are now commonplace in retail settings. However, these services require users to follow specific procedures and operate the systems according to predefined capabilities, which may exclude users who are unfamiliar with the systems or who require additional support. Although robots deliver essential services efficiently, their rigid designs limits their adaptability. By contrast, human service providers can flexibly tailor services by observing a customer’s condition through visual and auditory cues. For robots to offer more inclusive and user-friendly services, they must be capable of assessing user conditions and adapting their behaviors accordingly. Therefore, this paper proposes a control support system that analyzes user gaze behavior during interactions with smart appliances to provide context-aware support. Gaze data were collected using HoloLens 2, a mixed reality device, allowing the system to deliver information tailored to the user’s gaze direction. By providing an information support service through a robot based on an analysis of the user’s gaze, the user’s level of interest in the targeted environmental objects could be confirmed. Accordingly, a service that improves convenience and is tailored to the user could be provided. Finally, we discuss the effectiveness of the proposed human-centric robotic system through experiments.