Key-Points Based Visual Servo For Accurate Cross-Task Robot Manipulations

Key-points serve as critical visual features for robotic manipulation, yet their potential for guiding robots to handle objects and generalize across tasks remains underexplored. In this work, we propose a key-point based servoing framework that leverages a pre-trained Vision Transformer (ViT) to extract semantically rich key-points from images. Unlike prior works limited to known objects, a generic lightweight neural network is trained to directly map 2D key-point displacements to 3D Cartesian-space servo commands, bypassing explicit perspective geometry modeling. This network learns a unified 2D-to-3D correspondence that generalizes across diverse tasks and scenarios. Experiments in simulation and real-world setups demonstrate that our method achieves higher precision than optimization-based pose estimation and better generalization across unseen tasks. By bridging key-points with task-agnostic servo command prediction, this work advances feature-based manipulation research and offers practical solutions for logistics and service robots handling unstructured objects.