Policing Words with Machines: A Proof-of-Concept for LLM-Assisted Qualitative Analysis in Law Enforcement Research

Policing research has long utilized qualitative methods to understand the complex dynamics of law enforcement, yet these approaches often struggle with scalability and efficiency when applied to large datasets. This study explores the potential of artificial intelligence (AI), specifically large language models (LLMs), to enhance text classification and key theme extraction in qualitative policing research. Using open-ended survey responses from police officers regarding stress-relief activities, the study compared traditional hand-coding with three levels of LLM-assisted coding (basic, advanced, and tuned). Through iterative engineering of JSON-based prompts guiding the LLM, the tuned model achieved high agreement, sensitivity, and Jaccard similarity with human-coded results, significantly outperforming earlier iterations. Findings highlight the potential of LLMs to reduce time burdens on analysts and researchers while maintaining analytical integrity, though human oversight remains essential due to limitations in context comprehension, ethical considerations, and the evolving nature of qualitative data. This proof-of-concept demonstrates that, with proper tuning and supervision, LLMs can be a valuable tool for scalable and accurate qualitative analysis in sensitive domains like policing and other areas of social sciences.