EKF-enhanced hybrid visual servoing with deep perception for autonomous endoscope control in robotic surgery

In robot-assisted minimally invasive surgery (MIS), precise automatic control of the endoscope is crucial for maintaining optimal visualization. Regarding the additional burden imposed on surgeons due to manual adjustment of the field of vision, as well as the unstable perception caused by environmental noise during the surgical process, this paper proposes a Perception-Enhanced Hybrid Visual Servo (P-HVS) framework. Its core contribution lies in a unique architectural integration of perception and control at the system level, specifically designed to suppress measurement noise and provide a more stable and smoother perception data stream compared to the raw visual feedback during surgery. At the perception layer, the framework combines a deep learning model (YOLOX-STrans) with an Extended Kalman Filter (EKF) to form a processing pipeline that mitigates fluctuations in raw depth measurements from stereo vision, provide a more stable three-dimensional input for the control layer. The control layer then implements a speed-proportional blend of IBVS and PBVS based on this unified state. From an engineering perspective, this integration effectively bridges the gap between image-plane sensitivity and workspace accuracy while avoiding the command interruptions inherent in traditional discrete switching schemes. Quantitative simulation results on the dVRK platform demonstrate that the proposed hybrid servo framework excels in trajectory smoothness, tracking accuracy, and joint velocity stability, laying the foundation for autonomous control in the field of surgical robotics.