Can large language models be used to code text for thematic analysis? An explorative study

In practice, thematic analysis of text involves six stages, among which text coding is particularly cognitively demanding, labor-intensive, and time-consuming. This study investigates and compares the potential of two large language models (LLMs), namely ChatGPT-4 and OpenAI o1-preview, to perform text coding, with the goal of reducing the time and effort required by human researchers. Our results indicate that both models exhibit decreased coding comprehensiveness as document length increases, and both demonstrate low coding accuracy, primarily due to limitations in textual comprehension and reasoning. These findings highlight significant challenges in using LLMs to support thematic analysis, emphasizing the need for human oversight and rigorous validation to ensure analytic accuracy and validity.