Beyond Text or Image: A Comparative Study of Fine-Tuned VLM for Meme Hate Speech Detection

With the rapid growth of user-generated content on social media platforms, some user-generated hate speech memes and offensive content can lead to deterioration of the social media atmosphere and even real-world violence. This study applies a new visual language model IDEFICS3 to process multimodal meme datasets and compares the accuracy of deep learning algorithms (ResNet, Inception v3, LSTM, and Modern-Bert) that process only a single modality, and also compares the accuracy of a single-modality algorithm that undergoes feature fusion by methods such as the co-attention mechanism. After several sets of comparative experiments, it is concluded that the VLM model IDEFICS3 with the QLoRA fine-tuning strategy achieves the highest accuracy (.729) when tested on the benchmark dataset Hateful Memes for the multimodal hate speech detection task.