Face Recognition via Transformer Token Distillation

Despite the excellent performance of face recognition algorithms based on CNNs, these models typically rely on local receptive fields, which limit their ability to perform global modeling. To address the issue that traditional convolutions cannot fully extract global context, we proposes a transformer-based face recognition model FRT. By introducing the self-attention mechanism, FRT can overcome spatial distance limitations within an image and capture long-distance dependencies, thereby significantly enhancing its ability to perceive and capture global information. Considering that the original ViT may ignore inter-patch information, we make the tokens through a sliding window to generate overlapping patches. We also propose a teacher-student distillation strategy tailored for transformers, wherein a distillation token is introduced to ensure that the student learn prior knowledge from the CNN teacher, thereby inheriting the inductive biases of traditional convolutional models. Extensive experiments on the LFW, CALFW, CFP-FP, CPLFW and AgeDB-30 datasets demonstrate the effectiveness of the proposed FRT approach.