Bias Detection and Mitigation in Zero-Shot Spam Classification using LLMs

There is a growing number of scams through various communication mediums, including social media, phone calls and messages, search engine advertising, etc. Often these scams are realized via sending spam texts on any of these communication mediums, and therefore, prior research has investigated the task of spam classification to design information filtering systems. However, existing works have explored supervised machine learning techniques primarily, which suffer from the bottleneck of requiring large labeled datasets. Further, the studies are based on platform-specific data and lack critical analyses of biases in the predictive modeling behavior. In this paper, we propose a zero-shot spam classification task, which does not require any labeled data for model training in an unseen domain. We propose a novel method to leverage state-of-the-art large language models (LLMs) in natural language processing (NLP) for this task. We employ this method to analyze zero-shot performance on spam datasets across two communication platforms (YouTube and phone messages) while mitigating biases in the model behavior. Our experimental results demonstrate the strong performance of a LLM-based zero-shot classifier with a goal-oriented prompting strategy. The resulting classifier is platform-agnostic, shows less bias towards data with certain behavioral attributes (e.g., spam content with sadness emotion), and is efficient in minimizing both false positive and false negative errors. The application of this research can inform effective spam filtering systems to prevent scams prevalent in different communication media ultimately.