Analyzing the concordance and consistency of AI and human ratings in hospitality reviews

Purpose This study aims to explore the application of ChatGPT to analyze hotel guest satisfaction from online reviews. As online feedback plays a critical role in consumer decision-making in the hospitality industry, the research evaluates the accuracy and reliability of ChatGPT’s ratings compared to those of human raters and classic supervised machine learning classification techniques. Design/methodology/approach Using TripAdvisor reviews of five-star hotels, the authors use a structured two-phase study to assess both inter- and intra-rater reliability. Findings The results highlight distinct differences in rating behavior between artificial intelligence (AI) and human judges, with ChatGPT showing a tendency toward more moderate ratings. In addition, the authors observe a slight tendency for guests to overrate their experiences, supporting the literature on the subjective nature of online reviews. Despite these variations, ChatGPT shows significant agreement with guest ratings, especially when minor discrepancies are accounted for, suggesting its utility as a feedback analysis tool in the hospitality industry. This paper highlights ChatGPT’s ability to process and evaluate textual data and discusses the implications of using AI to improve review analysis processes in hospitality management. The authors advocate the incorporation of AI tools into customer feedback systems to augment human analysis and suggest future research to refine AI models for practical applications. Originality/value This study advances the understanding of AI’s role in hospitality management by demonstrating the practical application of ChatGPT for analyzing guest satisfaction through online reviews and providing a methodological framework for assessing the reliability of AI-generated content.