Representative Sample Augmented Hateful Memes Detection

In the digital media era, memes significantly influence emotional expression but can also convey hate through symbols or images, inciting harmful speech and division. The existing hateful memes detection methods are limited in their learning and generalization capabilities due to insufficient training data, which is caused by the difficulty of annotation. Therefore, we analyze the definition of memes from the perspective of social science and find that some representative visual features are repeatedly used as the carrier of hate metaphors in memes. Based on this, we propose a Representative Sample Augmented Hateful Memes Detection method (RSAHMD), which significantly improves the detection performance and generalizability of the model by introducing representative samples containing key visual features. Specifically, firstly, RSAHMD uses a representative sample retrieval method based on visual representative interpretation to select the most representative samples from existing datasets as auxiliary inputs, helping the model focus on metaphorical hate information. Secondly, we design an Adaptive Feature Weighting Module (AFWM) to dynamically adjust feature weights, enhancing the model's focus on key information while effectively reducing noise introduced from representative samples. Finally, we use the feedback mechanism to optimize the decision boundary of the model, and contrastive learning is applied to optimize the feature space distribution, reducing the interference of mislabeled samples and improving the ability of model to distinguish and retrieve complex samples. Experiments show RSAHMD improves performance and interpretability in detecting hateful memes. Its plug-and-play nature allows flexible application to different models and datasets, offering an efficient and transparent solution.