MREformer: When Vision Transformer Meets Convolution in the Field of Multimodal Relation Extraction

Relation extraction, as an important downstream task in the field of natural language processing, provides support for constructing knowledge graphs and semantic networks. Traditional relation extraction has focused on text-only domains. However, with the increasing prevalence of multimodal data such as image-text pairs, conventional relation extraction methods have shown limitations due to their inability to effectively process and utilize these types of data. As a result, extracting relationships from multimodal data becomes an increasingly important research direction in natural language processing and related fields. In this paper, we address the problem of relation extraction from multimodal data by proposing a Transformer-based model called MREformer. Our model consists of three main components. The first one is a visual encoder for image data, which is based on Vision Transformer and combines convolutional neural network to benefit from its advantages. Thus, it can reduce the loss during feature extraction and improve the effect. The second one is a text encoder for text data. We utilize Bert for this task which is a widely-used text encoder. The last one is a feature fusion module, which is based on M-Encoder of MKGformer. The M-encoder can reduce semantic information loss when fusing features from different feature spaces, thus it can achieve better results. Our model achieved an improvement of 1.42% in F1 score on the MNRE dataset compared to the baseline model, demonstrating the effectiveness of our approach.