Image-Text Integration: VAE-Enhanced GCN for Diagnostics

Medical multimodal fusion enhances diagnostic accuracy and comprehensiveness by integrating images, text, and other data types, leveraging the complementary information of different modalities. Traditional multimodal fusion techniques often prioritize the integration of various image modalities or simply add textual information, failing to fully exploit the connections and complementary nature between them. To address these limitations, we propose a novel multimodal fusion framework, HSGCN (Hidden State Graph Convolutional Network). This framework performs an initial fusion of image hidden states with text and a secondary fusion of image representations with diagnostic keywords, effectively utilizing the complementarity of multimodal data. We evaluate the performance of HSGCN against existing unimodal and multimodal models, and extensive experiments confirm that HSGCN is an effective multimodal fusion method that outperforms current approaches.