Fine-Tuned Large Language Models for Theory-Aligned Social Science Content Analysis: Evidence From Collaborative Learning Discourse

Theory-based content analysis is central to social science research, but coding context-dependent interactional discourse remains difficult to scale. This study examines whether fine-tuned large language models (LLMs) can support theory-aligned, episode-level coding of complex social interaction, using socially shared metacognitive regulation (SSMR) in collaborative learning discourse as a demanding empirical case. Using a human-coded dataset of 2,372 discourse episodes from undergraduate collaborative problem-solving, we adapted the large language model content analysis (LACA) framework through an AI-aligned codebook, prompt design, and supervised fine-tuning. Prompt-only and fine-tuned models were compared across episode-level classification and multilayer SSMR coding. Fine-tuned models substantially improved agreement with human coding and predictive performance relative to prompt-only baselines. Performance was strongest for episode-level classification and several higher-support coding layers, whereas errors remained concentrated in sparse, boundary-prone, and highly interpretive categories requiring social-intent reconstruction and temporally unfolding context. The findings extend recent work on LLM-assisted qualitative and content analysis by showing both the promise and limits of supervised, theory-aligned automation for multilayer social interaction coding. Overall, the study supports a human-centered approach to LLM-assisted content analysis, in which automated coding can support more stable coding tasks while human expertise remains essential for ambiguous, boundary-prone, and interpretively demanding judgments.

Authors

Institutions

Publication Details

Journal
Social Science Computer Review
Published
2026-11-12
DOI
https://doi.org/10.1177/08944393261502769
Primary Topic
Computational and Text Analysis Methods
Type
article
Field-Weighted Citation Impact
0.00

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Fine-Tuned Large Language Models for Theory-Aligned Social Science Content Analysis: Evidence From Collaborative Learning Discourse

Yoonhee Shin, YoungKwan Son, Haengkyung Lee, Minyoung Kim
Social Science Computer Review
Computational and Text Analysis Methods
article

Fine-Tuned Large Language Models for Theory-Aligned Social Science Content Analysis: Evidence From Collaborative Learning Discourse

Yoonhee Shin, YoungKwan Son, Haengkyung Lee, Minyoung Kim
article en

Abstract

Theory-based content analysis is central to social science research, but coding context-dependent interactional discourse remains difficult to scale. This study examines whether fine-tuned large language models (LLMs) can support theory-aligned, episode-level coding of complex social interaction, using socially shared metacognitive regulation (SSMR) in collaborative learning discourse as a demanding empirical case. Using a human-coded dataset of 2,372 discourse episodes from undergraduate collaborative problem-solving, we adapted the large language model content analysis (LACA) framework through an AI-aligned codebook, prompt design, and supervised fine-tuning. Prompt-only and fine-tuned models were compared across episode-level classification and multilayer SSMR coding. Fine-tuned models substantially improved agreement with human coding and predictive performance relative to prompt-only baselines. Performance was strongest for episode-level classification and several higher-support coding layers, whereas errors remained concentrated in sparse, boundary-prone, and highly interpretive categories requiring social-intent reconstruction and temporally unfolding context. The findings extend recent work on LLM-assisted qualitative and content analysis by showing both the promise and limits of supervised, theory-aligned automation for multilayer social interaction coding. Overall, the study supports a human-centered approach to LLM-assisted content analysis, in which automated coding can support more stable coding tasks while human expertise remains essential for ambiguous, boundary-prone, and interpretively demanding judgments.

Social Science Computer Review
Yonsei University (KR), Hanyang University (KR)
Hanyang University
Computational and Text Analysis Methods
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon