Global and Local Similarity Aggregation for Face Sketch Recognition

The presence of a heterogeneous gap between images and sketches, stemming from the abstraction of sketches and the concreteness of images, is the fundamental factor contributing to the challenge of the Face Sketch Recognition (FSR) task. Existing works primarily focus on a solution that first achieves heterogeneous synthesis and then proceeds with homogeneous matching while neglecting the synthesized images lack the valid information available in the original images for cross-view alignment, resulting in discrepancies between the generated and real images. To address this issue, we propose an innovative multi-view representation learning framework, which solves FSR in one step, with the constraint of feature similarity to capture cross-view invariant information. Specifically, we introduce quaternion data to enhance data richness and feature discriminability by leveraging the randomness of generated samples. Furthermore, we design a novel Similarity Aggregation Network (SAN) that imposes constraints on the local feature similarity and global semantic similarity between quaternion data, thereby aligning local cross-view features and constructing compact global representations. Therefore, the proposed SAN effectively bridges the heterogeneous gap between sketches and images in the feature space by directly extracting sufficient and effective cross-view invariant information. Comprehensive experiments on five commonly used datasets demonstrate the superiority of our approach over state-of-the-art methods.