Fed-urlBERT: Client-Split Federated BERT for Malicious URL Detection

Malicious URL attacks are becoming more domain-specific, for example, phishing tactics varying greatly between sectors, such as education and e-commerce. Therefore, cross-domain collaboration and knowledge sharing are essential for developing effective solutions to combat these threats. However, such collaboration is often hindered by privacy concerns and business sensitivities. Federated learning addresses these challenges by enabling multi-client collaboration without direct data exchange. Unfortunately, if highly expressive Transformer models are used, clients may face intolerable computational burdens, and the exchange of weights could quickly deplete network bandwidth. In this paper, we propose Fed-urlBERT, a federated learning method for malicious URL detection using pretrained Transformers. It leverages split learning to partition the Transformer encoder into client-side embeddings and server-side encoding. In this setup, only generate initial embeddings from local data, while the server handles the computationally intensive encoding. Our approach addresses privacy concerns and cross-domain collaboration needs in malicious URL defense while minimizing the client computational burden. We employ two aggregation strategies, FedAvg and FedALA, with the latter designed to mitigate client heterogeneity by integrating both local and global model information during each training round. We evaluate our approach across three distinct data distribution scenarios, including Independent and Identically Distributed (IID data) and two types of Non-IID data settings. The experimental results demonstrate that our method consistently achieves performance comparable to that of centralized models. Significantly, under balanced data conditions, our federated model also demonstrates a notable reduction in the False Positive Rate (FPR), highlighting a potential regularization benefit of the federated approach. The source code is accessible at https://github.com/Davidup1/FedURLBERT.