Investigation of the Inter-Rater Reliability between Large Language Models and Human Raters in Qualitative Analysis

Qualitative analysis is typically limited to small datasets because it is time-intensive.Moreover, a second human rater is required to ensure reliable findings.Artificial intelligence tools may replace human raters if we demonstrate high reliability compared to human ratings.We investigated the inter-rater reliability of state-of-theart Large Language Models (LLMs), ChatGPT-4o and ChatGPT-4.5-preview, in rating audio transcripts coded manually.We explored prompts and hyperparameters to optimize model performance.The participants were 14 undergraduate student groups from a university in the midwestern United States who discussed problemsolving strategies for a project.We prompted an LLM to replicate manual coding, and calculated Cohen's Kappa for inter-rater reliability.After optimizing model hyperparameters and prompts, the results showed substantial agreement ( > 0.6) for three themes and moderate agreement on one.Our findings demonstrate the potential of GPT-4o and GPT-4.5 for efficient, scalable qualitative analysis in physics education and identify their limitations in rating domain-general constructs.