DeepMEL: A multi-agent collaboration framework for multimodal entity linking

• We propose DeepMEL, a multi-agent framework based on Role-Specialized, whichfuses visual and context information through collaboration between LLMs andLVMs. To our knowledge, this is the first multi-agent approach applied to MEL. • The modality conversion alignment strategy for cross-modal representationalignmentthrough two paths: using a Context Summary Prompt with LLMs to generatefine-grained descriptions, and a Visual Q¥&A Prompt with LVMs to extract structuredimage representations. • We propose an Adaptive Iteration strategy that dynamically optimizes candidatesbycombining tool-based retrieval with LLM-based semantic reasoning. • We propose a unified prompt that reformulates MEL into a structuredcloze-styletask, simplifying input parsing and improving LLMs’ semantic understandingof multimodal disambiguation. • Substantial experiments on many datasets validate the superiority of our method Multimodal Entity Linking (MEL) aims to associate textual and visual mentions with entities in a multimodal knowledge graph. Despite its importance, current methods face challenges such as incomplete contextual information, coarse cross-modal fusion, and the difficulty of jointly large language models (LLMs) and large visual models (LVMs). To address these issues, we propose DeepMEL, a novel framework based on multi-agent collaborative reasoning, which achieves efficient alignment and disambiguation of textual and visual modalities through a role-specialized division strategy. DeepMEL integrates four specialized agents, namely Modal-Fuser, Candidate-Adapter, Entity-Clozer and Role-Orchestrator, to complete end-to-end cross-modal linking through specialized roles and dynamic coordination. DeepMEL adopts a dual-modal alignment path, and combines the fine-grained text semantics generated by the LLM with the structured image representation extracted by the LVM, significantly narrowing the modal gap. We design an adaptive iteration strategy, combines tool-based retrieval and semantic reasoning capabilities to dynamically optimize the candidate set and balance recall and precision. DeepMEL also unifies MEL tasks into a structured cloze prompt to reduce parsing complexity and enhance semantic comprehension. Extensive experiments on five public benchmark datasets demonstrate that DeepMEL achieves state-of-the-art performance, improving ACC by 1 %-57 %. Ablation studies verify the effectiveness of all modules.