Incorporating Artificial Intelligence in Qualitative Research: Exploring the Role of ChatGPT in Thematic Analysis

Abstract Thematic analysis is a form of qualitive analysis performed to identify patterns within text-based datasets such as open-ended responses. While thematic analysis is used extensively in medical education research, it has several limitations, such as subjective interpretation by graders and the time required to manually code responses in large datasets. There is potential to overcome many of these challenges with the use of Artificial Intelligence (AI) platforms, such as the free to use large language model, ChatGPT. The goal of this study was to evaluate whether AI can be used in thematic analysis to replace manual graders as the gold standard. The dataset used in this study was related to first year medical students’ thoughts and feelings regarding the act of cadaveric dissections. Three different methods were used to instruct the AI to grade the responses, and each method was repeated three times. Various measures related to precision and accuracy were compared, both within the repeated tests using AI and between the AI generated results and those obtained by manual coders. Results show that Method 3 had greater accuracy and agreement with the manual coders, but less precision compared to the other two methods. All methods had an agreement greater than 80%. These findings demonstrate that AI has promise in being used for thematic analysis, but the method used to instruct the AI has a strong influence on the results. When using AI for thematic analyses, it is imperative to carefully document and refine the methodology as well as still incorporate some human oversight to ensure accurate results.