Medxchat: A Unified Multimodal Large Language Model Framework Towards CXRS Understanding and Generation

Multimodal Large Language Models (MLLMs) have shown promise in general image tasks, but their application in medical imaging remains limited. This study explores MLLMs for Chest X-Rays (CXRs) through MedXChat, a unified framework enabling seamless interaction for tasks such as text report generation, visual question-answering (VQA), and Text-to-CXR generation. Our MLLM, which uses natural language as input, breaks task boundaries and simplifies medical professional training to the maximum extent by allowing diverse tasks within a single environment. To understand CXRs, we employ visual encoders (e.g.,ViT) and LLMs (e.g., mPLUG-Owl), which we refine using visual adapters and delta-tuning. For CXR generation, we introduce an innovative synthesis approach that leverages the instruction-following capabilities of the Stable Diffusion (SD) architecture to produce fine-grained medical images without additional parameters. Through comprehensive experiments, our model demonstrates exceptional cross-task adaptability, showcasing proficiency across all three tasks.