Few-Shot Prompting, Full-Scale Confusion: Evaluating Large Language Models for Humor Detection in Croatian Tweets

Humor detection in low-resource languages is hampered by cultural nuance and subjective annotation.We test two large language models, GPT-4 and Gemini 2.5 Flash, on labeling humor in 6,000 Croatian tweets with expert gold labels generated through a rigorous annotation pipeline.LLM-human agreement ( = 0.28) matches human-human agreement ( = 0.27), while LLM-LLM agreement is substantially higher ( = 0.63).Although concordance with expert adjudication is lower, additional metrics imply that the models equal a second human annotator while working far faster and at negligible cost.These findings suggest, even with simple prompting, LLMs can efficiently bootstrap subjective datasets and serve as practical annotation assistants in linguistically underrepresented settings.