Using large language models to complement humans for the coding of social media interactions between science teachers

Abstract This methodological paper investigates the viability of using Large Languages Models (LLMs) for the use of qualitative coding of posts to Facebook groups for science teachers. Categories for the coding, and the corresponding prompt for the LLMs, were engineered using an iterative process on progressively larger data sets, whilst checking consistency with human expert coders at each step. Three different versions of OpenAI’s LLMs were used; GPT-3.5 Turbo, GPT-4 Turbo, and GPT-4o, to also investigate the difference in performance between the different versions. The LLMs performed consistently well in qualitative coding, showing moderate to good agreement with human coders. This performance remained strong even when applied to datasets outside the prompt’s original design. Agreement of the LLMs with the human coders was slightly below human–human agreement levels for the two older LLMs, whereas the most recent LLM, GPT4o, achieved an inter-coder reliability that matched that of the human coders. This study proposes several possible strategies for the integration of LLMs into qualitative coding workflows, as well as offering suggestions to improve LLM performance through prompt engineering.