Applying large language models to spam detection in the Kazakh low-resource language setting

In this paper, we test the efficacy of multilingual transformer-based large language models for spam classification tasks on the low-resource Kazakh language. Due to the lack of sufficient labeled data for spam emails written in the Kazakh language, we evaluate the ability of modern multilingual models for generalized tasks, especially the gains obtainable through supervised learning. In this analysis, we test a range of popular models such as bert-base-multilingual-cased, distilbert-base-multilingual-cased, and xlm-roberta for spam detection that are either trained for zero-shot tasks or through parameter-efficient supervised learning using LoRA. The experimental findings show that the zero-shot accuracy of multilingual LLMs is inconsistent, especially for the less common spam class, where the models tend to lack the ability to generalize because of the underrepresented Kazakh corpus. Yet, the models achieve considerable accuracy after being trained on a labeled corpus, where all models achieve accuracy above 0.99, especially XLM-R, which achieves a macro-F1 value of 0.99 and substantially reduces false negatives. A closer examination of these misclassified instances shows that common linguistic patterns, such as marketing necessities, financial jargon, or the conversational tone, play a significant role in their misclassification. The experimental results indicate that a limited amount of annotated instances for the Kazakh language, leveraging the efficiency of LoRA-based fine-tuning, can be effectively combined to improve the multilingual models' performance even on security-related tasks.