State Representation Learning for Visual Servo Control

We propose a method for visual servo-control of robots using images from an uncalibrated camera that constructs compact state representations of the robot’s configuration and uses transition dynamics learned from collected execution traces to compute control velocities to reach a desired goal state identified directly by its image. The key step of the proposed method is the estimation of a homography transform between the image positions of distinct keypoints belonging to the robot in the current image and those in a reference image, which can be done quickly and robustly even when not the same set of keypoints is observed at each time step, making it robust to noise and variations in illumination. The estimated homography is then used to represent the robot configuration as the image coordinates of a minimal number of virtual points moving with the robot. The method was verified experimentally for planar motion of a fully actuated manipulator arm as well as an underactuated mobile robot with a nonholonomic constraint.