Comparing language models and machine learning classifiers for detecting humorous vaccine misinformation on social media

Abstract Misinformation embedded in humor on social media can be difficult to detect and mitigate. This study investigates the effectiveness of traditional machine learning (ML) models, pre-trained neural language models, and large language models (LLMs) in identifying humor-laden misinformation related to COVID-19 vaccines on Twitter. A dataset of 1,500 tweets was randomly sampled from a corpus of 17,945 humorous vaccination disclosures, with human coders annotating the presence of misinformation. We compared logistic regression, a linear support vector classifier (SVC), RoBERTa, and OpenAI’s GPT-4o with a zero-shot and a few-shot prompt, together with a keyword baseline. The few-shot GPT-4o prompt had the highest mean recall and F1 score for the types of misinformation indicated in the prompt and had higher recall and F1 than a keyword baseline built from those same terms. RoBERTa’s accuracy, precision, and F1 score did not differ significantly from GPT-4o’s, and its performance changed little with smaller training sets. Logistic regression and SVC showed lower accuracy, recall, and F1 scores than RoBERTa and the few-shot GPT-4o prompt. This study underscores the potential of LLMs to augment human efforts in detecting complex forms of misinformation on social media and highlights the importance of model selection based on task requirements.