Augmented-Reality Based Assistive Systems for Critical Medical Procedures

Augmented reality (AR) holds significant potential to enhance human performance in high-stakes, safety-critical domains where errors can have severe consequences. By overlaying digital information directly into the user’s field of view, AR can support rapid decision-making, maintain situational awareness, and improve precision under critical conditions. However, the complexity of such domains and the limited scope of interdisciplinary research have hindered the development of suitable AR-based systems for these specific use cases. This dissertation investigates how AR can be harnessed to improve human performance in tasks that require substantial cognitive effort, precise motor control, and rapid, complex decision-making under pressure. Critical medical procedures, in which these challenges are inherently present, serve as the primary context for investigation. Through an interdisciplinary approach that integrates computer science and medical practice, the work develops methods designed to support clinicians in high-demand environments. To this end, this thesis first focuses on building foundational knowledge about the requirements for viable AR systems that can enhance human performance. It then explores the design space and impact of AR for procedural and motor skill acquisition by examining nurse training in a cardiopulmonary resuscitation scenario. Following this, it investigates AR systems and their role in decision-making and cognitive support during high-risk procedures, with a specific focus on their integration into pancreatic tumor removal surgery. Through clinical trials, it further examines the downstream, real-world impacts of such systems on surgical performance and patient outcomes. Finally, it explores context-aware AR designs using large language models to provide additional cognitive support and decision-making capabilities. Drawing on all of these investigations, the thesis distills design implications for the future development of AR systems in high-stakes medical environments.