Text-Guided Hierarchical Visual Prefix Network for Multimodal Relation Extraction

The purpose of Multimodal Named Entity Recognition (MNER) and Multimodal Relation Extraction (MRE) is to identify meaningful entities and their relationships in text with visual information. This paper proposes a Text-Guided Multilevel Visual Fusion Network (Tg-MVFNet) to extract better and utilize relevant visual features. First, the Vision Transformer model encodes global and local object images to obtain multilevel visual features that align better with text features. Second, a text-guided dynamic gating mechanism is designed to generate visual prefixes, adjusting visual weights dynamically based on text content to ensure relevant information is emphasized. Finally, cross-modal feature fusion is achieved using attention to visual prefixes, allowing each BERT layer to access multilevel visual features for more informed predictions. We conducted experiments on three benchmark datasets for MNER and MRE tasks. Integrating our text-guided dynamic gating module with existing models achieves state-of-the-art performance, confirming its effectiveness.