Feature Enhancement for Multi-modal Entity Alignment Transformer

Multi-modal entity alignment (MMEA) aims to identify equivalent entities across different multi-modal knowledge graphs. In these graphs, entities are enriched with information from various modalities, such as text, images, and numerical data, making the alignment task both challenging and crucial for improving knowledge graph quality. Current MMEA algorithms typically encode entity information separately for each modality using corresponding encoders, and then integrate these representations through various modal fusion strategies. However, these methods often fail to fully exploit the multi-modal information of entities. To address this issue, we propose a feature-enhanced multi-modal entity alignment transformer (FEMEAT). FEMEAT enhances entity attribute information by incorporating modal distribution data, which captures the inherent distribution of different modalities for each entity. This inclusion allows the model to have a richer understanding of entity characteristics across modalities. Additionally, FEMEAT utilizes an Optical Character Recognition (OCR) model to extract and incorporate textual information from images. By integrating this text extracted from images, the model can better utilize the visual modality, enhancing its ability to understand and process multi-modal information. Furthermore, FEMEAT employs a multi-head cross-modal attention (MHCA) mechanism for modal fusion to achieve comprehensive multi-modal entity representation. This mechanism enables the model to attend to different modalities simultaneously and learn a detailed representation of entities by considering the interactions between modalities. The multi-head cross-modal attention mechanism facilitates a nuanced understanding and integration of multi-modal data. Experimental results demonstrate that our model achieves state-of-the-art (SOTA) performance across various training scenarios. The code and datasets used in this study can be accessed at https://github.com/zewenD/FEMEAT.