Multi-strategy enhanced privacy protection for multimodal data in federated learning

Federated learning protects privacy while training on distributed clients, facing challenges from Non-IID data and sensitive information. This paper introduces a multi-strategy privacy-preserving framework using Local Differential Privacy (LDP) for client privacy, a Genetic Algorithm (GA) for robust aggregation of noisy updates, and Generative Adversarial Networks (GANs) for personalized data augmentation, particularly for multi-modal data. LDP on clients and GA on the server enhance privacy and aggregation. Client-specific GANs generate synthetic data reflecting local data characteristics, including modalities, addressing Non-IID issues and data scarcity. Experiments on MNIST, CIFAR-10, and the multimodal Camelyon17 show our framework better balances privacy and model utility. Results highlight GA's effectiveness in noisy aggregation and GANs' capability in augmenting data across modalities for improved performance.