Human vs. LLM Labeling Consistency: A Comparative Study of Sentiment Annotation in Open-Ended Survey Responses

This study investigates the extent to which human annotators agree on sentiment classification of open-ended survey responses compared to large language models (LLMs). We will measure inter-annotator agreement among multiple human raters and between human raters and LLMs using Krippendorff’s Alpha as the primary metric and exact match percentage as a secondary descriptive statistic. The research will help assess whether LLMs can achieve levels of subjective judgment comparable to human raters in sentiment annotation tasks.