Anatomy-Aware Enhancement and Cross-Modal Disease Representation Fusion for Medical Report Generation

The objective of medical report generation is to convert medical images into structured text reports. The limitations of existing methods primarily include the lack of efficient local modeling and fusion algorithms, as well as severe data bias in medical datasets. To address these issues, we propose a novel method for report generation based on multi-feature fusion, comprising Anatomy-Aware Fusion Enhancement (AAFE) and Cross-modal Disease Representation Fusion (CDRF). This method first localizes specific anatomical structures and then models the interdependencies among anatomy-aware tokens through AAFE, achieving deep integration of global and anatomical tokens. Simultaneously, CDRF dynamically fuses cross-modal clinical indications, visual features, and disease states to obtain a high-order representation rich in visual and disease information, enabling the generator to generate fine-grained medical reports based on rich multi-feature embedding. Furthermore, we introduce a cross-modal cosine similarity loss mechanism to further promote the extraction of disease text features with anatomy-aware information. Experimental results on the IU-Xray and MIMIC-CXR datasets demonstrate that our method outperforms existing advanced models in the medical report generation task and exhibits outstanding performance.