LLMs vs. humans in sarcasm detection on german soccer tweets: The impact of domain knowledge and voting type

Abstract With LLMs increasingly used as conversational assistants, this study compares the sarcasm detection performance of five LLMs and 323 humans in German, conducting a binary classification task of 100 soccer-related tweets. The analysis includes metrics of individual vote and majority vote for both groups, examining how voting type affects classification accuracy. Human participants were further grouped by their level of soccer knowledge to assess the effect of domain knowledge on detection accuracy. Among the LLMs, Mistral Large achieved the highest accuracy (83%), nearly matching the human average (83.3%). Significant performance differences were observed among LLMs and between LLMs and humans, with the largest gap between Gemini 1.0 Pro (64%) and the human majority vote (97%). Majority vote significantly improved human classification accuracy by 13.7 percentage points, and domain knowledge was positively correlated with sarcasm classification accuracy. Humans also performed similarly across sarcastic and non-sarcastic tweets, whereas LLMs frequently misclassified tweets containing negative lexical markers as sarcastic. These results highlight the importance of ensemble strategies and domain knowledge in sarcasm detection tasks and underscore the need for further research into polarity-related classification biases in LLMs.