Get What You See: Using Images to Manipulate Robots

How to give models the ability to understand non-direct image instructions? In this paper a discussion was done using an end-to-end network with a two-layer architecture, selecting images from different viewpoints of the object as the target instructions, together with different instruction feature extraction methods. After testing, using the pre-trained classification network as the feature encoder can make the model understand the instruction correctly up to 89.6%, and the average localization error is reduced to 24.633 mm. And the model was given a very high generalization ability by our method.