Federated Learning Meets Recursive Self-Distillation: A Scalable Malware Detection Framework for IoVs

This paper proposes an integrated approach called FL-RSD, leveraging the key advantages of Federated Learning (FL) and Recursive Self-Distillation (RSD) for malware detection in the Internet of Vehicles (IoV). The proposed FL-RSD framework enhances model generalization, mitigates overfitting to non-IID data, and improves adaptability to new malware variants. The RSD process iteratively transfers knowledge from a teacher model to a lightweight student model, reducing model complexity and communication overhead while preserving detection accuracy. Experimental results have confirmed that FL-RSD achieves significant performance improvements over baseline models in terms of malware detection accuracy and adversarial robustness. FL-RSD attains a malware detection accuracy and an average adversarial robustness scores over 92%, outperforming Federated Proximal, FedNova, and Hierarchical FL. The improvements range from 3% to 48% for detection accuracy, and 6% to 81% for the adversarial robustness score. Additionally, FL-RSD demonstrates a minimal memory footprint with a final global model size of 373.17 KB and maintains knowledge retention with an Average Catastrophic Forgetting Score of 93.66%. These results confirm that FL-RSD offers a lightweight, efficient, and scalable solution for malware detection in IoV.