Adapter With Textual Knowledge Graph for Zero-Shot Sketch-Based Image Retrieval

Zero-shot sketch-based image retrieval (ZS-SBIR) refers to retrieving natural images with similar contours and semantics using query sketches in zero-shot scenarios. Existing works mainly focus on using teacher networks to mitigate catastrophic forgetting of models. They aim to retain the knowledge learned by pre-trained models on large-scale datasets, thereby improving the zero-shot generalization ability of models. However, naively using teacher networks is difficult to effectively mitigate catastrophic forgetting and leads to overfitting risks in some cases. In addition, previous methods ignore the explicit utilization of structural knowledge and fail to guide the model to fully mine the structural knowledge of category semantic information, thereby neglecting the learning of inter-class relations. To address these issues, we propose an adapter with textual knowledge graph (ATKG) for ZS-SBIR. Specifically, we employ a dual constraint of prompt learning strategies and teacher networks to suppress catastrophic forgetting and reduce overfitting risks. Additionally, to effectively learn inter-class relationships, we introduce an adapter with textual knowledge graph. This method treats category semantic information and visual features as nodes, computing their edges using cosine similarity. Subsequently, a graph convolutional network (GCN) is used to mine the structural knowledge between nodes, further effectively learning relationships among different categories. Extensive experiments on the Sketchy, TU-Berlin, and QuickDraw datasets demonstrate that our proposed method achieves better retrieval performance than state-of-the-art competitors. Specifically, on Sketchy split1, TU-Berlin and QuickDraw datasets, mAP@all is improved by 3.2%, 4.3% and 5.1% compared to IVT or CLIP-all, respectively. On the more challenging Sketchy split2 dataset, mAP@200 is improved by 2% compared to Dr.CLIP.