TS-SLink: a two-stage span linking network for joint multimodal entity-relation extraction

Multimodal Named Entity Recognition (MNER) and Multimodal Relation Extraction (MRE) are fundamental and core tasks in the field of information extraction. Recently, Joint Multimodal Entity-Relation Extraction (JMERE) has emerged as a unified task aimed at enhancing the completeness and consistency of extracted information. However, existing JMERE methods often suffer from redundant annotations when processing multi-word entities, leading to label conflicts and entity boundary ambiguity. This reliance on implicit feature interactions leads to weak interaction strength and limited interpretability. To address these issues, we propose TS-SLink, a two-stage span linking network that unifies entity recognition and relation extraction as a multi-class classification task by constructing entity spans from boundary tokens to reduce redundant labeling. In the first stage, we integrate pixel-level and semantic image features into textual representations through a multimodal encoder incorporating semantic anchor contrastive learning and multi-granularity feature fusion, providing rich contextual semantic support for the model. Based on the span linking, we design an NER tagger and an RE tagger to jointly predict the entity-relation quintuples. In the second stage, to capture the semantic constraints between the NER and RE predictions, we introduce an entity-relation type transition matrix, which encodes prior knowledge about the semantic transferability between entity and relation types, enhancing task coordination and model interpretability. Experimental results on the JMERE dataset demonstrate the effectiveness of TS-SLink, which achieves state-of-the-art performance.