FedCMPTrans: Federated Fine-Tuning of Cross-Modality Projection Large Models for Medical Domain

Utilizing deep learning models in the medical domain could contribute greatly to healthcare and further applications. For visual tasks such as medical image segmentation, the ability to understand the image and extract vital information is very important. Therefore, large models that show great visual ability in the general domain are introduced. To better harness those pretrained large models in the medical domain, many researchers choose to adapt large models to medical downstream tasks by fine-tuning. However, compared to general domain data, medical data may be harder to acquire and have privacy problems. To this end, Federated Learning (FL) is introduced to utilize distributed data and meanwhile meet the privacy demands. To be specific, we proposed FedCMPTrans, a framework and method to federatedly fine-tune pretrained large models to the medical domain, mainly medical Visual Question Answering (VQA) task. We choose a cross-modality projection model architecture to maintain the ability to generate text, and thus could directly carry out VQA task instead of convert it into classification task. Following previous work, we fine-tune the cross-modality projector model which bridges the visual and text modalities. Additionally, to better utilize the ViT model, a learnable visual prompt is used. Furthermore, to cope with the influence of non-IID data in the FL scenario, we use a contrastive loss term to help with the optimization. Experiments show that FedCMPTrans could acquire good performance on medical VQA tasks while maintaining text generative workflow.

Authors

Institutions

Publication Details

Journal
ACM Transactions on Multimedia Computing Communications and Applications
Published
2026-11-21
DOI
https://doi.org/10.1145/3860321
Primary Topic
Multimodal Machine Learning Applications
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

FedCMPTrans: Federated Fine-Tuning of Cross-Modality Projection Large Models for Medical Domain

Zhiwei Song, Ruhui Ma, Hongjian Shi, Yicheng Di et al.
ACM Transactions on Multimedia Computing Communications and Applications
Multimodal Machine Learning Applications
article

FedCMPTrans: Federated Fine-Tuning of Cross-Modality Projection Large Models for Medical Domain

Zhiwei Song, Ruhui Ma, Hongjian Shi, Yicheng Di, Haibing Guan, Yuan Liu, Qiwei Yu, Yuhan Qiu, Xiaohui Zhong
article en

Abstract

Utilizing deep learning models in the medical domain could contribute greatly to healthcare and further applications. For visual tasks such as medical image segmentation, the ability to understand the image and extract vital information is very important. Therefore, large models that show great visual ability in the general domain are introduced. To better harness those pretrained large models in the medical domain, many researchers choose to adapt large models to medical downstream tasks by fine-tuning. However, compared to general domain data, medical data may be harder to acquire and have privacy problems. To this end, Federated Learning (FL) is introduced to utilize distributed data and meanwhile meet the privacy demands. To be specific, we proposed FedCMPTrans, a framework and method to federatedly fine-tune pretrained large models to the medical domain, mainly medical Visual Question Answering (VQA) task. We choose a cross-modality projection model architecture to maintain the ability to generate text, and thus could directly carry out VQA task instead of convert it into classification task. Following previous work, we fine-tune the cross-modality projector model which bridges the visual and text modalities. Additionally, to better utilize the ViT model, a learnable visual prompt is used. Furthermore, to cope with the influence of non-IID data in the FL scenario, we use a contrastive loss term to help with the optimization. Experiments show that FedCMPTrans could acquire good performance on medical VQA tasks while maintaining text generative workflow.

ACM Transactions on Multimedia Computing Communications and Applications
Jiangnan University (CN), SingHealth (SG), Shanghai Jiao Tong University (CN), Bombardier (Canada) (CA)
Multimodal Machine Learning Applications
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon