Expanding automated LLM coding of question type: Tags and ‘do you know’ questions

Abstract Purpose Large language models (LLMs) are increasingly used in place of manual coding to classify questions in forensic interviews for the purposes of training and evaluation. However, models that have achieved high reliability with manual coders can generally only distinguish among broad question categories. We examined whether an automated question type coding model can reliably distinguish not only among four core prompt types (invitations, wh questions, option‐posing questions, non‐questions) but also two previously unexplored question categories associated with problematic response tendencies: tag questions and ‘do you know’ questions. Methods An LLM (RoBERTa) was trained on questions ( N = 351,920) from forensic interviews ( n = 1435) and trial testimony ( n = 416) involving 3‐ to 17‐year‐old alleged victims of child sexual abuse. Results The machine achieved high reliability with the manual codes (95% agreement; K = .94). Sensitivities for the two novel question categories (tags and ‘do you know’ questions) were 96% and 94%, respectively. Specificities approached 100% for both question types. Conclusions Automated question type coding is a highly accurate alternative to manual coding that can reliably discern among question types, even among less common question types such as tag questions and ‘do you know’ questions.