DMoLE: Dynamic Mixture of LoRA Experts for Spam Email Detection

Email safety is crucial for network security and person privacy. Traditional machine learning methods can achieve good performance on detecting spam emails, but these methods have poor generalization ability. Nowadays, pretrained language models significantly improve the generalization ability due to the powerful semantic understanding capabilities. However, with the larger and larger models scale, fine-tuning models on spam email detection is expensive. Many Parameter-Efficient Fine-Tuning methods are proposed to address this issue. But they still have weaknesses falling short of full fine-tuning. In this paper, we propose Dynamic Mixture of LoRA Experts (DMoLE) to address all above challenges. The DMoLE combines LoRA modules as experts with a gate network to scale up LoRA and we propose a Dynamic Expert Allocation algorithm to mitigate the expert redundancy by pruning less important components of experts. Moreover, DMoLE can achieve the same performance with full model fine-tuning on spam email detection with few trainable parameters, and it also has good generalization capability.