Imitation learning-based Direct Visual Servoing using the large projection formulation

Today robots must be safe, versatile, and user-friendly to operate in unstructured and human-populated environments. Dynamical system-based imitation learning enables robots to perform complex tasks stably and without explicit programming, greatly simplifying their real-world deployment. To exploit the full potential of these systems it is crucial to implement closed loops that use visual feedback. Vision permits to cope with environmental changes, but is complex to handle due to the high dimension of the image space. This study introduces a dynamical system-based imitation learning for direct visual servoing. It leverages off-the-shelf deep learning-based perception modules to extract robust features from the raw input image, and an imitation learning strategy to execute sophisticated robot motions. The learning blocks are integrated using the large projection task priority formulation. As demonstrated through extensive experimental analysis, the proposed method realizes complex tasks with a robotic manipulator. • A dynamical system-based imitation learning approach for direct visual servoing. • Uses deep-learning-based perception modules and IL to execute sophisticated motions. • Learning blocks are integrated using the large projection task priority formulation. • Extensive experiments show that our method realizes complex manipulation tasks. • The proposed method outperforms baselines in multiple real-world robot tasks.