HOW CAN MULTIMODAL DEEP LEARNING DETECT VIOLENCE IN LOW-RESOURCE SINHALA SOCIAL MEDIA POSTS?

Social media is an integral aspect of daily life, yet it also serves as an avenue for disseminating violent content. Image-based posts, in particular, serve as powerful vehicles for such content, prompting efforts to automatically detect and combat this issue. While studies predominantly focus on English content, fewer explore languages with limited digital resources, such as Sinhala. Our research pioneers the classification of Sinhala image posts based on both textual and visual features and introduced a foundational work in the context of a low resource language, Sinhala by adapting standard models for this underexplored domain. As the first step, we introduced a new annotated dataset of Sinhala image posts, with extracted textual and visual components. Furthermore, we employed a customized Tesseract OCR and the EAST library for automatic extraction of these components. After pre-processing the textual data, we utilized models, including LSTM, GRU, 1D CNN, and ensemble models combining 1D CNN with GRU, LSTM, BiGRU, and BiLSTM, alongside word2vec embeddings. Additionally, we incorporated XLM-R, a multilingual transformer-based model, to enhance textual classification. For visual analysis, pre-trained models such as ResNet, AlexNet, ViT, and EfficientNet were employed. Finally, we employed an SVM-based late fusion multimodal model for classification. Our approach achieved 95% accuracy, surpassing single-modality results. This marks significant progress in addressing violent content in Sinhala image posts on social media. Although architecture employs standard models, adopting these models to Sinhala language and the creation of a baseline dataset marks a significant contribution