Sketch Face Recognition Method Based on Local-Global Adapter

The objective of sketch-based face identification is to match a target individual’s facial features from a collection of photographs using a sketched portrait as the search query. The method based on Contrastive Language-Image pre-training (CLIP) brings matched sketch and optical images closer together through learnable text tokens, therefore improving recognition performance. However, the existing sketch face datasets are relatively small. Training the CLIP with a large number of parameters on these datasets makes model overfit, leading to suboptimal performance. To address the above issue, we propose a sketch face recognition method based on Local-Gobal Adapter (LGAdapter). The method uses a transformer to capture facial details, and we segment the features through H windows, thus limiting the self-attention mechanism to localized windows, which better captures detailed information. In order to enhance the extraction of more comprehensive representational characteristics, we incorporate a streamlined Graph Convolutional Network (GCN) module subsequent to the local module. In addition, we propose a two-stage training strategy to fully utilize the LGAdapter to obtain more accurate visual features. In the first phase, only text tokens are optimized; in the second phase, only the LGAdapter is fine-tuned. The outcomes of our experiments reveal that our approach surpasses existing leading-edge methods across all three widely recognized datasets, namely UoM-SFGS, CUFSF, and PRIP-VSCG.