Fine-Tuned Large Language Models for Theory-Aligned Social Science Content Analysis: Evidence From Collaborative Learning Discourse
Theory-based content analysis is central to social science research, but coding context-dependent interactional discourse remains difficult to scale. This study examines whether fine-tuned large language models (LLMs) can support theory-aligned, episode-level coding of complex social interaction, using socially shared metacognitive regulation (SSMR) in collaborative learning discourse as a demanding empirical case. Using a human-coded dataset of 2,372 discourse episodes from undergraduate collaborative problem-solving, we adapted the large language model content analysis (LACA) framework through an AI-aligned codebook, prompt design, and supervised fine-tuning. Prompt-only and fine-tuned models were compared across episode-level classification and multilayer SSMR coding. Fine-tuned models substantially improved agreement with human coding and predictive performance relative to prompt-only baselines. Performance was strongest for episode-level classification and several higher-support coding layers, whereas errors remained concentrated in sparse, boundary-prone, and highly interpretive categories requiring social-intent reconstruction and temporally unfolding context. The findings extend recent work on LLM-assisted qualitative and content analysis by showing both the promise and limits of supervised, theory-aligned automation for multilayer social interaction coding. Overall, the study supports a human-centered approach to LLM-assisted content analysis, in which automated coding can support more stable coding tasks while human expertise remains essential for ambiguous, boundary-prone, and interpretively demanding judgments.
Authors
- Yoonhee Shin (ORCID: https://orcid.org/0000-0001-6684-3404)
- YoungKwan Son (ORCID: https://orcid.org/0009-0001-2620-2335)
- Haengkyung Lee
- Minyoung Kim (ORCID: https://orcid.org/0009-0005-7092-4895)
Institutions
- Yonsei University (KR)
- Hanyang University (KR)
Publication Details
- Journal
- Social Science Computer Review
- Published
- 2026-11-12
- DOI
- https://doi.org/10.1177/08944393261502769
- Primary Topic
- Computational and Text Analysis Methods
- Type
- article
- Field-Weighted Citation Impact
- 0.00
Funders
- Hanyang University