A Visual Question Answering-Driven Fine-Grained Cross-Modal Alignment Framework for Radiological Report Generation

Automatically generating diagnostic reports from radiological images is crucial for clinical practice but faces challenges in fine-grained image-text alignment and controllable content generation. Existing methods, primarily focusing on direct image-to-report alignment, struggle with capturing subtle visual features for complex medical descriptions and are prone to generating hallucinatory information. To address these limitations, we propose a novel visual question answering-driven fine-grained cross-modal alignment framework for radiological report generation, shifting from traditional image-report alignment to the visual question answering paradigm. Our approach employs an LLM-based data transformation strategy to convert long-text medical reports into multiple questionanswer pairs, targeting specific anatomical regions, symptomatic abnormalities, or diagnostic conclusions. This significantly enhances the supervision of image-text alignment and the learning of local semantics. We also utilize a multimodal generative language model architecture with visual prefix embedding for VQA training, enabling precise capture of detailed features and deep semantic understanding of medical images. Experimental results on public datasets demonstrate that our method outperforms mainstream state-of-the-art models.