Privacy Preserving in Federated Learning with Transformer Models

Privacy protection in federated learning (FL) is a critical challenge, especially with the integration of Transformer models for improved learning. Traditional FL methods struggle with privacy threats such as model inversion attacks and gradient leakage, relying on differential privacy and secure aggregation, which often cause accuracy loss and computational overhead. To overcome these limitations, we propose a hybrid model combining Swin Transformer and Federated Transformer to enhance privacy, efficiency, and scalability. The Swin Transformer extracts hierarchical representations from decentralized data while preserving local privacy and feature integrity, whereas the Federated Transformer ensures secure client collaboration through advanced self-attention mechanisms. We validate our approach using the CYBRIA dataset, achieving 98.5% accuracy, a 97.8% F1-score, and a 35% privacy leakage reduction. Additionally, our token-based aggregation technique decreases communication overhead by 15%. These results demonstrate the robustness, security, and effectiveness of our framework for real-world cybersecurity and healthcare applications.