Higher agreement, different errors: A preregistered out-of-sample evaluation of error-informed instructions for LLM coding of teacher talk moves

Coding instructions for large language models (LLMs) are often revised from the models' errors against human codes, but rarely frozen and tested on new data, other models and single codes. In four preregistered studies on teacher talk moves, an optimizer model revised an LLM coder's instructions, and we tested the frozen result for transfer, costs and remaining errors. On locked test lessons, agreement with human codes (Cohen's kappa) rose from .67 to .76, below the preregistered target of .81 and not clearly above a revised manual. Unchanged, the instrument raised kappa by .08 to .22 on new teachers across five model families. Per move, kappa lay between .72 and .91, against .91 to 1.0 between the human annotators. Precision on rare moves rose for every coder, whereas their recall fell clearly for two coders. Higher kappa did not give more accurate move shares per teacher. In exploratory control runs, a generic instruction reached one fifth of the gain and rules written without human codes two fifths. Setting aside the 10% of sentences on which three models disagreed raised kappa on the rest to .92. Error-informed instructions transfer, but their errors depend on the coder and matter differently by use. Preprint. This version has not been peer reviewed. Supplementary Material S1, the software and all data and scripts are in the replication packages (https://doi.org/10.5281/zenodo.23911470).

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-18
DOI
https://doi.org/10.5281/zenodo.24349945
Primary Topic
Computational and Text Analysis Methods
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Higher agreement, different errors: A preregistered out-of-sample evaluation of error-informed instructions for LLM coding of teacher talk moves

Simon Filler
5 links
Zenodo (CERN European Organization for Nuclear Research)
Computational and Text Analysis Methods
preprint

Higher agreement, different errors: A preregistered out-of-sample evaluation of error-informed instructions for LLM coding of teacher talk moves

Simon Filler
preprint en

Abstract

Coding instructions for large language models (LLMs) are often revised from the models' errors against human codes, but rarely frozen and tested on new data, other models and single codes. In four preregistered studies on teacher talk moves, an optimizer model revised an LLM coder's instructions, and we tested the frozen result for transfer, costs and remaining errors. On locked test lessons, agreement with human codes (Cohen's kappa) rose from .67 to .76, below the preregistered target of .81 and not clearly above a revised manual. Unchanged, the instrument raised kappa by .08 to .22 on new teachers across five model families. Per move, kappa lay between .72 and .91, against .91 to 1.0 between the human annotators. Precision on rare moves rose for every coder, whereas their recall fell clearly for two coders. Higher kappa did not give more accurate move shares per teacher. In exploratory control runs, a generic instruction reached one fifth of the gain and rules written without human codes two fifths. Setting aside the 10% of sentences on which three models disagreed raised kappa on the rest to .92. Error-informed instructions transfer, but their errors depend on the coder and matter differently by use. Preprint. This version has not been peer reviewed. Supplementary Material S1, the software and all data and scripts are in the replication packages (https://doi.org/10.5281/zenodo.23911470).

Zenodo (CERN European Organization for Nuclear Research)
TU Dortmund University (DE)
Quality education
Computational and Text Analysis Methods
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon