Speech-Based Path Refinement in Robot- Assisted Surgery through Virtual Obstacles

Abstract Mobile surgical robots must operate within the spatial constraints of the operating room (OR) while adhering to strict safety and integration requirements. While robots progress towards autonomy, it remains important that humans always maintain control over the robots’ actions. For example, when planning paths to navigate through the OR, it is possible that the robot overlooks an obstacle or lacks contextual information, such as predicting humans’ intent to move or sudden environmental changes, including accidents. Therefore, the path planning must be both precise and modifiable by the medical staff. Existing systems for modifying robot paths often disrupt the surgical workflow, require extensive training, or, in the case of speech-controlled systems, offer only limited interaction vocabularies. This paper proposes a speech-based system to refine the mobile robot path using natural language as input. Our core contribution is a natural language interface that enables users to modify robot paths by converting verbal instructions into virtual obstacles, thereby reshaping the planned path.We demonstrate the feasibility of this system in a simulated ophthalmic surgery scenario. We evaluated the system on 10 different surgical environment occupancy maps and three different voice commands per map, resulting in a success rate of 96.67%. Path comparisons between human-drawn and automatically generated paths confirm the intended behavior of the system.