Development of human-collaborative robots to perform daily tasks based on multimodal vital information with cybernics space

Due to the increasing employment of people with disabilities, support for the elderly, and childcare needs, enhancing the independence and freedom across generations and spaces is necessary. This study aimed to develop a human-collaborative robot using multimodal vital information as input with cybernics space, which is fused "human" and "Cyber/Physical Space," and confirm its feasibility experimentally. The robot allows the user to operate it via gaze and bio-electrical signals (BES), reflecting the user's intentions, and seamlessly transition among three modes (i.e., assistant, jockey, and ghost). In the assistant mode, the user collaborates with the robot in the physical space using a system that includes a head-mounted display (HMD) for gaze measurement, BES measurement unit, personal mobility system, and an arm-hand system. The HMD can be flipped up and down for hands-free control. The BES measurement unit captures leaked weak signals from the skin surface, indicating the user's voluntary movement intentions, which are processed by the main unit to generate control commands for the various actuators. The personal mobility system features omni-wheels for tight turning, and the arm-hand system can handle payloads up to 500 g. In the jockey mode, the user remotely operates a small mobile base with a display and camera, moving it through the physical space. In the ghost mode, the user navigates and inputs commands in a virtual space using a smart key and remote-control device integrated with IoT and wireless communication. The switching of each control mode is estimated using the BES from the user's upper arm, gaze direction, and position, thereby enabling movement, mobility, and manipulation without physical body movement. In basic experiments involving able-bodied participants, the macro averages of recall, precision, and F score were 1.00, 0.90, and 0.94, respectively, in the assistant mode. The macro averages of recall, precision, and F score were 0.85, 0.92, and 0.88, respectively, in the ghost mode. Therefore, the human-collaborative robot utilizing multimodal vital information has feasibility for supporting daily life tasks, contributing to a safer and more secure society by addressing various daily life challenges.