Multi-modal personalized federated learning with adaptive differential privacy for medical image classification and a privacy-preserving approach

Deep learning on medical images classification intervention needs to use large data on multi-institutional datasets but privacy laws inhibit sharing of data (GDPR, HIPAA). Federated Learning (FL) facilitates collaborative training without data transfer; until now, the known methods can only address privacy, personalisation, and accuracy not at the same time in a multi-modal environment. We present MM-PFL-ADP, a framework that combines Vision Transformer (ViT) based multi-modal feature extraction in four new elements: (i) privacy budget allocation (independent of number of samples): Fisher information-based adaptive per-parameter privacy budget allocation ([Formula: see text]); (ii) personalisation masks: dynamic KL divergence based personalisation masks; (iii) respect The framework gives formal client-level [Formula: see text]-DP guarantees on transmitted gradient updates, in [Formula: see text] simulated medical institutions. On the MRI-MS dataset, MM-PFL-ADP achieves [Formula: see text] accuracy (95% CI: 96.9-[Formula: see text]) at [Formula: see text], outperforming FedAvg ([Formula: see text]) and DP-FedAvg ([Formula: see text]) by large margins ([Formula: see text]). The framework is [Formula: see text] faster than FedAvg (47 vs. 85 rounds), has [Formula: see text] less total communication and keeps [Formula: see text] accuracy in case of extreme heterogeneity in data ([Formula: see text]). The probability of membership inference attack has decreased to 52.1 which was close to the random baseline ([Formula: see text]). MM-PFL-ADP shows that the concepts of privacy, personalisation, and accuracy are synergistic, but not oppositional to federated medical AI. The single-system Fisher information framework greatly simplifies the hyperparameter tuning problem and can meet formal privacy criteria. Before being deployed, prospective validation against the performance of expert radiologists is desired.