Freehand Interaction With Visual Control and Haptic Feedback in Electromagnetically Assisted Interventional Surgery

Internet of Medical Things (IoMT) technology has significantly helped surgeons perform complex clinical procedures, including interventional and endoscopic surgeries. However, surgeons face challenges in quickly becoming proficient with some IoMT devices. This difficulty stems from the fact that IoMT devices are not commonly used or familiar in their daily work and lives. To address this issue, we propose a novel interactive framework for IoMT, named freehand interaction, which includes visual control and haptic feedback. Our idea is to allow surgeons to operate surgical instruments, such as needles, capsules, and catheters with their bare hands and regular experience. At the same time, identical instruments in the surgical environment mirror the surgeon’s actions through visual control. The interactive forces encountered in the surgical environment are quantitatively communicated to the surgeon through the hand-held instruments. Our IoMT framework achieves both visual control and haptic feedback using electromagnetic mechanisms, ensuring mid-air freehand manipulation and contactless remote actuation. We perform different tasks in suspended, liquid, and in-vitro environments. The hand-held and mirroring instruments show high similarity and correlation, even within different electromagnetic systems and confined workspaces. Tracking accuracy, response time, and haptic forces quantified by a mechanical gauge are satisfactory. Our algorithm processes raw video streams and maintains efficiency even in scenarios with partial hand occlusion and various types of image noise. In summary, this work contributes to improving visual intuition and haptic immersion in IoMT applications.