Applying machine learning text classification methods and large language models for data collection of mentions of child maltreatment

Background: Research on the efficacy of interventions to prevent or alleviate child maltreatment (CM) has been impeded by a lack of structured data from child protection services (CPS). Significant case knowledge is encapsulated within text documents authored by caseworkers, but much of this information is only available in an unstructured manner. Recently, innovative machine learning (ML) approaches have emerged utilizing large language models (LLMs). These LLMs have the potential to analyze detailed texts and extract structured data on multifaceted CPS case factors. Objective: This study aims to examine the applicability and accuracy of ML text classification methods and LLMs for extracting structured data from casework reports. We seek to classify reports on whether they mention a child experiencing physical abuse, a child exhibiting aggressive behaviors towards others, a child experiencing sexual abuse, or a child witnessing domestic violence. Methods: All CPS casework reports dating from 2008 to 2022 in Switzerland’s largest canton were analyzed (N=29’947). Text segments based on child maltreatment keywords were extracted. A random sample was annotated for training/testing purposes. The test dataset was annotated by a second expert reviewer, and inter-rater reliability analysis was conducted. The performance of different ML approaches on correctly classifying texts was evaluated, and the optimal model was applied on the full data set. Validation analysis of the findings was implemented. Results: The best-performing LLM, Mixtral-8x-7B, classified mentions of CM with an accuracy of 87%. Validation analysis suggests that automated classification was as accurate as expert human classification. Conclusions: Our study yields supporting evidence for applying ML methods and LLMs to effective and accurate text classification in child maltreatment research. Further research on multi-label classification, bias testing and validation involving caseworkers is needed. Takeaway Points: There is evidence that large language models can be applied effectively and accurately to extract structured data from casework reports in child maltreatment research. This opens up possibilities for using these data in subsequent analyses. There is room for further improvement, e.g., the development of multi-label classification methods is advised. Further validation efforts are also needed which should examine the validity of classifications from the perspective of caseworkers authoring the reports and the possible occurrence of algorithmic biases.