A Cross-Modal Interactive Memory Network Based on Fine-Grained Medical Feature Extraction for Radiology Report Generation

Radiology report generation is an essential task in the medical field, which aims to automate the generation of medical terminology descriptions of radiology images. However, this task currently suffers from several problems: 1) existing methods need to manually build knowledge graphs or templates (consuming time and effort) to introduce medical or prior knowledge to assist in report generation; 2) previous models cannot handle the problem of data bias well (anomaly reports and anomaly descriptions make up only a tiny portion of the dataset), causing the models to ignore the learning of anomaly descriptions easily; 3) existing approaches cannot robustly supervise the model, resulting in incomplete and inconsistent reports being generated. To address these issues, we propose a cross-modal interactive memory network based on fine-grained medical feature extraction. In our model, we design a cross-modal interactive memory network to automatically store and remember the required medical text knowledge and use this medical knowledge to help generate reports. Furthermore, we design an abnormal medical knowledge enhancement module to enhance the learning of abnormal fine-grained knowledge through the interaction of disease topics and their states to interact with text features. In addition, we design a cross-modal joint semantic loss unit to reduce semantic differences between different features and improve the visual representation ability of the model. We experimented and evaluated our model on MIMIC-CXR and IU-Xray datasets to compare with other baseline models.