Amh3MD multimodal dataset and deep learning model for information disorder detection in social media

Misinformation, disinformation, and mal-information have all become significantly more common due to the quick expansion of social media platforms. These forms of abusive content often appear in multimodal formats such as text, images, and memes, making their detection more challenging, particularly for low-resource languages such as Amharic. This study proposes a multimodal deep learning model for detecting information disorder by integrating textual and visual features through early fusion. A new multimodal dataset (Amh3MD) containing 8053 labeled samples sourced from the most popular social media platforms. and grouped into 4 classes: disinformation, misinformation, mal-information, and normal content. Textual features were extracted using contextual embedding models, including DistilBERT, while visual features were extracted using deep convolutional and transformer-based architectures such as ResNet, EfficientNet, and Vision Transformer (ViT). Several multimodal deep learning models, including CNN, LSTM, BiLSTM, CNN-LSTM, and transformer-based architectures, were evaluated using performance evaluation metrics. Experimental results demonstrate that early fusion approaches, which integrate visual and textual features, substantially enhance classification performance. Among the evaluated models, the CNN-BiLSTM multimodal model achieved the best performance in the CNN/LSTM group with an accuracy of 0.63, an F1-score of 0.58, and a recall of 0.57. In transformer-based experiments, the EF-ResNet50-mBERT-Attention model achieved the highest overall performance with an accuracy of 0.68 and an F1-score of 0.67, demonstrating the effectiveness of attention-based multimodal fusion. The results present the importance of contextual text embeddings and advanced visual representations for multimodal information disorder detection. This study contributes to the field by introducing the first publicly available Amharic multimodal information disorder dataset and proposing an effective fusion-based deep learning model for multi-class classification. The findings provide valuable insights for developing scalable and context-aware models to combat information disorders and harmful content in low-resource languages. Future research should focus on scalable, multilingual, multimodal transformer and vision-language models to improve performance and to develop a real-time social media monitoring system for detecting information disorder.