Multimodal Aggressive Meme Classification using Bidirectional Encoder Representations from Transformers

Aggressive memes, often characterized by harmful or provocative content, pose a challenge for detection across multiple modalities. To address this, we developed the MAC dataset, categorizing memes into non-aggressive, political, religious, gendered, and other forms of aggression. We evaluated various textual models, including CNN, BiLSTM, various BERT, and visual models, such as VGG16, VGGG19, ResNet50, and Vision Transformer, employing a decision fusion technique for multimodal analysis. Bangla BERT-1 achieved the highest f<inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</inf>-score (0.74) among textual models, while Vision Transformer led visual models (f<inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</inf>-score: 0.68). Our multimodal approach combining Bangla BERT and Vision Transformer outperformed individual models with a weighted f<inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</inf>-score of 0.76, excelling in political and religious aggression classes though underperforming in non-aggressive and gendered categories.