FedCMPTrans: Federated Fine-Tuning of Cross-Modality Projection Large Models for Medical Domain
Utilizing deep learning models in the medical domain could contribute greatly to healthcare and further applications. For visual tasks such as medical image segmentation, the ability to understand the image and extract vital information is very important. Therefore, large models that show great visual ability in the general domain are introduced. To better harness those pretrained large models in the medical domain, many researchers choose to adapt large models to medical downstream tasks by fine-tuning. However, compared to general domain data, medical data may be harder to acquire and have privacy problems. To this end, Federated Learning (FL) is introduced to utilize distributed data and meanwhile meet the privacy demands. To be specific, we proposed FedCMPTrans, a framework and method to federatedly fine-tune pretrained large models to the medical domain, mainly medical Visual Question Answering (VQA) task. We choose a cross-modality projection model architecture to maintain the ability to generate text, and thus could directly carry out VQA task instead of convert it into classification task. Following previous work, we fine-tune the cross-modality projector model which bridges the visual and text modalities. Additionally, to better utilize the ViT model, a learnable visual prompt is used. Furthermore, to cope with the influence of non-IID data in the FL scenario, we use a contrastive loss term to help with the optimization. Experiments show that FedCMPTrans could acquire good performance on medical VQA tasks while maintaining text generative workflow.
Authors
- Zhiwei Song (ORCID: https://orcid.org/0000-0002-1990-0645)
- Ruhui Ma (ORCID: https://orcid.org/0000-0001-9592-8490)
- Hongjian Shi (ORCID: https://orcid.org/0000-0003-0743-7806)
- Yicheng Di (ORCID: https://orcid.org/0000-0003-3802-2080)
- Haibing Guan (ORCID: https://orcid.org/0000-0002-4714-7400)
- Yuan Liu (ORCID: https://orcid.org/0000-0002-2576-1426)
- Qiwei Yu (ORCID: https://orcid.org/0009-0001-0713-6854)
- Yuhan Qiu (ORCID: https://orcid.org/0009-0006-1603-3263)
- Xiaohui Zhong (ORCID: https://orcid.org/0009-0007-2722-875X)
Institutions
- Jiangnan University (CN)
- SingHealth (SG)
- Shanghai Jiao Tong University (CN)
- Bombardier (Canada) (CA)
Publication Details
- Journal
- ACM Transactions on Multimedia Computing Communications and Applications
- Published
- 2026-11-21
- DOI
- https://doi.org/10.1145/3860321
- Primary Topic
- Multimodal Machine Learning Applications
- Type
- article
- Field-Weighted Citation Impact
- 0.00