Named entity recognition and coreference resolution using prompt-based generative multimodal

Named Entity Recognition (NER) and Coreference Resolution (CR) are both of pivotal significance in the domain of natural language processing. The accurate detection of pronouns and noun phrases has been shown to have a direct impact on cross-sentence and even cross-modal semantic understanding. However, existing methods often suffer from inadequate semantic fusion in multimodal scenarios, leading to issues such as text-image mismatches and incorrect referential resolution. The present paper proposes a multimodal generation framework that integrates object detection with prompt-based strategies. First, a prompt mechanism is employed to embed entity types into the visual channel using object detection results and hierarchical features extracted by a visual Transformer. This design enhances multimodal NER. Then, the PGMCR model is constructed, which aligns the pronouns and noun phrases identified by NER with the image regions based on prompt information to achieve generative multimodal co-reference resolution. The experimental findings on the public CIN dataset demonstrate that PGMNER attains an F1 score of 96.55% in the NER task, which is notably higher than that of representative multimodal models. PGMCR attains a CoNLL F1 score of 62.53% on the CR task, thereby surpassing existing state-of-the-art methods. These results provide validation of the effectiveness of the proposed framework in cross-modal named entity recognition and co-reference resolution, thus providing new insights for multimodal semantic understanding in complex scenarios.