Can an AI Chatbot Reliably Assess the Reporting Quality of Health-Related Qualitative Research? A Pilot Study

Generative AI (genAI) is increasingly being integrated into critical appraisal and knowledge synthesis of research, with growing discourse on its value for qualitative research. Moreover, ChatGPT covers a major portion of the market share and is widely used by researchers and academics. However, a major gap remains in genAI’s ability to assess the reporting quality of qualitative research studies. This pilot study aimed to determine whether GPT-5 could reliably assess the reporting quality of health-related qualitative research studies. The Standards for Reporting Qualitative Research (SRQR) checklist was used as a framework for completing both human and AI chatbot assessments. The SRQR is a widely used validated checklist for enhancing the reporting of qualitative research. Overall, 68 peer-reviewed studies were selected for testing, comparing human SRQR assessments with GPT-5 assessments. Findings revealed a wide variability in agreement between human and GPT-5 assessments, with crude agreement ranging between 60 and 100% and interrater reliability ranging between 0 and 0.85 across checklist items. Simpler, objective checklist items such as study purpose were often correctly identified and had stronger agreement compared to nuanced items. The model struggled with the following SRQR items leading to an “Unclear” response: research design/paradigm (41%), researcher reflexivity (41%), data processing (27%), and trustworthiness (46%). These preliminary results suggest that GPT-5 may struggle with tasks requiring nuance and interpretation of qualitative research, which is rich in complex methods and differing worldviews. Currently, GPT-5 and other AI chatbots should be used with caution and alongside human oversight when assessing the reporting quality of qualitative research within public health and healthcare.