Stable Vision-Based Robot Kinematic Control With Deep Learning-Based Oriented Object Detector

Recent advances in machine learning and deep learning have significantly enhanced robot control by improving object detection and visual feature extraction. However, ensuring theoretical guarantees of stability and convergence in learning-enabled control systems remains a major challenge. In this paper, we propose a vision-based control framework that integrates a deep learning oriented-object detector with a Lyapunov-stable servo control law. The proposed method ensures provably stable convergence of the robot end-effector or its grasped object's pose to a desired camera image region for both eye-in-hand and eye-to-hand configurations. Unlike existing deep learning based visual servoing methods, which either lack formal stability guarantees or ignore object orientation control, our approach incorporates object orientation into the control loop through a region-based method using quaternion representation and formally guarantees stability. We validated our framework on a 6-DoF UR5e manipulator performing cup insertion and centering tasks, demonstrating accurate and stable control in both camera setups.