Natural language-labeled keypoint graphs for industrial object localization

Abstract As industries increasingly move towards automation, the ability to visually localize objects and accurately estimate their pose, including rigid and non-rigid transformations, becomes critical. Conventional pose estimation methods are constrained to specific object categories, whereas category-agnostic approaches remain limited in accuracy and in their ability to handle composite objects with repeated and non-unique components. In this paper, we present a general framework for object localization based on keypoint graphs, where nodes represent keypoints and edges encode pairwise relations. Both nodes and edges are annotated with natural language descriptions. We introduce a novel neural network architecture that accepts natural language-labeled keypoint graphs as prompts and predicts the image coordinates of graph nodes. Furthermore, we make several industrial datasets publicly available and demonstrate that our method substantially outperforms existing methods. Additionally, we achieve strong performance on the public MP-100 benchmark while offering greater flexibility in representing and localizing complex objects.