Entity Label-Guided Graph Fusion Multi-Modal Named Entity Recognition

Multimodal Named Entity Recognition (MNER) is a task that leverages multimodal information (such as text and images) to identify named entities within social media text. Traditional MNER methods primarily rely on simple interactions between text annotations and visual features, thus overlooking the specific correspondence between text and visual objects. Additionally, irrelevant visual noise may interfere with the final recognition results. Therefore, this paper proposes a Entity label-guided Graph Fusion Multi-modal Named Entity Recognition approach(ELGF), which utilizes pre-defined entity label information from input text as a bridge. Firstly, entity label detection tasks are employed to obtain entity label information. Then, the entity label information is utilized as a bridge between the two modalities to construct a multimodal interaction graph. This graph is inputted into a graph neural network, where attention and gate mechanisms are applied to interactively fuse multimodal information. Finally, a Conditional Random Field (CRF) decoding is used to predict the final MNER label sequence. Extensive experimental results demonstrate that compared to mainstream methods, the proposed model achieves competitive recognition accuracy on public datasets.