Multimodal Social Relation Extraction Based on Hypergraph Attention Neural Networks

Multimodal social relation extraction requires effective feature fusion to recognize relationships across various targets. However, existing work often struggles to model the finer correlations among various modalities and ignores the higherorder complex relationships between multimodal information. To overcome current limits, a multimodal relationship extraction method based on hypergraph attention neural networks is proposed. The method can encode the higher-order data correlations in the hypergraph structure, obtain the higher-order complex relationships between multimodal information, and reduce the noise generated in the process of hypergraph encoding and enhance the correlation with text semantics through the cross-diffusion attention mechanism. Ultimately, it is combined with the original unimodal text and visual input to enhance the inference ability. Experimental results on three character social relationship datasets, Dream of the Red Chamber (DRC-TF), Water Margin (OM-TF), and Four Classics (FC-TF), in which TF indicates the inclusion of both textual and facial image data, clearly show the advantages of our and proposed methodology.