Interactive Instance Search: User-Centered Enhanced Image Retrieval with Learned Perceptual Image Patch Similarity

Instance searches pertain to the identification of specific objects or scenes within a dataset that correspond to a given query image. The existing research primarily concentrates on improving the accuracy of machine-recognized instances, frequently neglecting the pivotal role of human–computer interaction. As a result, effectively searching for instances that align with user preferences continues to pose a substantial challenge. In this paper, we introduce an intuitive and efficient instance search method that incorporates human–computer interaction. Specifically, our proposed interactive instance search system includes tools that enable users to directly highlight specific instances of interest within the query image. Furthermore, we propose the use of learned perceptual image patch similarity to effectively bridge the semantic gap between low-level features and high-level semantics. Contrary to conventional metrics, such as cosine similarity, which rely on pixel-level or superficial feature comparisons, we employ deep neural networks to model perceptual differences in a hierarchical manner. The experimental results demonstrate that our approach surpasses traditional methods in terms of similarity-matching accuracy and exhibits robust performance on datasets such as Oxford5k and Paris6k.