Holo4Care: a MR framework for assisting in activities of daily living by context-aware action recognition

Abstract The evolution of virtual and augmented reality devices in recent years has encouraged researchers to develop new systems for different fields. This paper introduces Holo4Care, a context-aware mixed reality framework designed for assisting in activities of daily living (ADL) using the HoloLens 2. By leveraging egocentric cameras embedded in these devices, which offer a close-to-wearer perspective, our framework establishes a congruent relationship, facilitating a deeper understanding of user actions and enabling effective assistance. In our approach, we extend a previously established action estimation architecture after conducting a thorough review of state-of-the-art methods. The proposed architecture utilizes YOLO for hand and object detection, enabling action estimation based on these identified elements. We have trained new models on well-known datasets for object detection, incorporating action recognition annotations. The achieved mean Average Precision (mAP) is 33.2% in the EpicKitchens dataset and 26.4% on the ADL dataset. Leveraging the capabilities of the HoloLens 2, including spatial mapping and 3D hologram display, our system seamlessly presents the output of the action recognition architecture to the user. Unlike previous systems that focus primarily on user evaluation, Holo4Care emphasizes assistance by providing a set of global actions based on the user’s field of view and hand positions that reflect their intentions. Experimental results demonstrate Holo4Care’s ability to assist users in activities of daily living and other domains.