Privacy-Preserving Medical Image Diagnosis using Federated Learning and Vision Transformers

Deep learning model training is a central approach, which poses risks to patient data privacy and hampers data sharing for medical image classification, essential for computer-assisted diagnosis. In this study, a privacy-preserving medical image classification system is presented that combines Vision Transformers (ViTs) with Federated Learning (FL) for the classification of four categories of white blood cells: eosinophils, lymphocytes, monocytes, and neutrophils. The framework uses model training on local data in three simulated clients and uses Federated Averaging (FedAvg) to merge model updates without sharing raw images. A conventional Convolutional Neural Network (CNN) and a centralized ViT are used as baseline models. The preliminary results indicate that the centralized ViT can attain an accuracy of 58.50% and a macro-F1 score of 51.82%, which is better than the CNN baseline (53.50% accuracy). The overall performance of the federated ViT can be seen as good with an accuracy score of 49.50% and a recall score of 96.00% for the class of monocytes, which suggests good class-specific recognition and, simultaneously, room for improvement at the overall level. An additional enhancement that can help prevent information leakage from model updates is proposed as differential privacy. Additional rounds of communication, larger sets of training data, hyperparameter optimization and formal guarantees of privacy will be explored in further work. This is a framework to give a basis for co-operative privacy-aware medical image classification.