One-Shot Visual Detection of Phishing Resources with CLIP ViT and Contrastive Learning

We propose a method for detecting phishing sites based on analyzing visual similarity with trusted web pages. The developed original approach utilizes the CLIP ViT model trained on its own dataset using triplet loss to generate page embeddings in a compact vector space. The key feature is the introduction of a multimodal representation of embeddings and the use of one-shot classification to detect new phishing resources without the need to retrain the model. The proposed approach combines visual and textual representation of web pages to improve robustness against HTML code and URL changes. A combined learning strategy including batch-hard triplet loss with soft margin and embedding regularization through MSE minimization between image and text representations is introduced. A series of experiments are conducted on a large set of real web pages containing legitimate and phishing sites. The obtained results demonstrate the high accuracy of the method, surpassing traditional approaches in terms of robustness to page modifications. The possibility of integrating the proposed system into the infrastructure of automated web threat monitoring is demonstrated.