Bilateral human teleoperation : a mixed reality system for remote ultrasound

Remote guidance of ultrasound exams and other medical procedures has the potential to transform healthcare delivery around the world. Currently, healthcare access is highly inequitable, with patients in remote communities being forced to wait weeks or travel hundreds of kilometres for even routine ultrasound exams. While remote care can help, current methods are complex, expensive, and impractical in the case of robotic telehealth, while video teleguidance is imprecise and inefficient. This dissertation proposes a novel tele-ultrasound system that combines mixed reality (MR), telecommunication, and haptics to bridge the gap between robotics and video guidance. In human teleoperation, the follower robot in a conventional bilateral teleoperation system is replaced by a novice human follower who wears an MR headset. Aligning their ultrasound probe with a virtual rendering of the same probe, they precisely track the motions of a remote expert. The expert views the live ultrasound stream and a video of the patient, and moves a haptic device to input their desired motion. In addition to the general human teleoperation concept, new force sensing and pose tracking methods are described in this thesis that enable stable and transparent four-channel teleoperation to give the expert a realistic and intuitive experience. The teleoperation system is modeled and various approaches are evaluated. Different rendering schemes are also tested to optimize the human computer interaction, and a high-performance communication system is developed and tested in various network conditions. Furthermore, a full telerobotic ultrasound system is developed to explore the effects of time delays in ultrasound teleoperation and to compare robotic and human teleoperation. Several sets of patient studies with the human teleoperation system are presented to evaluate its performance in a practical setting. These include experiments in the laboratory, at a local hospital, and in a remote community with the expert and follower separated by 754km. The combination of conceptual and technical contributions of this dissertation together with the patient tests constitute an argument for a new approach to tele-health that can benefit countless people and improve health equity worldwide.