MSLSNet: A combination of multi-task self-supervised learning and Swin transformer network for face and keypoint detection in thermal images

Accurate tracking in thermal images is critical for applications such as surveillance, authentication , and healthcare diagnostics, which rely on reliable face and facial keypoint detection. However, the scarcity of annotated thermal datasets poses a major challenge, reducing detection performance. Therefore, we introduced a multi-task self-supervised learning approach with a hybrid model, combining convolutional neural networks and transformers, tailored to enhance feature extraction in thermal images, even with limited data. Our convolutional-based model employed four pretext tasks—gender and subject classification, rotation prediction, and image inpainting . These tasks enabled learning intricate facial structures without face and keypoint annotations, generalizing effectively to diverse and unseen data. For face detection, we pretrained the model on these tasks and used fully connected. For keypoint detection, we used the pretrained model alongside a Swin transformer in a parallel configuration where these models worked simultaneously to leverage local and global features for improved accuracy. Fully connected layers were used to capture spatial relationships for accurate feature mapping in keypoint detection. By combining convolutional networks for local features and transformers for global context, the models demonstrated strong generalization. The models achieved an intersection over union (IoU) of 0.92 for face detection and a normalized mean error of 1.36 for keypoint detection, representing a 35.24 % improvement over the best-reported results. Additionally, our face detection algorithm identified 80 % of faces with a high IoU (>0.9), compared to 20 % by prior methods on a Thermal dataset. These results highlight the robustness of our approach, making it suitable for real-world applications.