Preregistration: Study 4 of 'Searching the hidden inventory' - can LLM coders show where they are likely wrong? (TalkMoves Subset 2)

Test of two signals for sentences that LLM coders of teacher talk moves probably get wrong: disagreement among Claude Sonnet 5.5, GPT-6 Luna and Gemini 4 Argon, and the stated confidence of Claude Sonnet 5.5. A frozen annotation harness codes 97 new passages from 8 teachers of TalkMoves Subset 2. Nothing is optimized. The primary measure is Cohen's kappa on the sentences not flagged. Confirmatory hypotheses test each signal against no flagging and compare the two directly, with cluster bootstrap over teachers and transcripts and Holm correction. Registered before the passages were drawn.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-02
DOI
https://doi.org/10.5281/zenodo.23896103
Primary Topic
Computational and Text Analysis Methods
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Preregistration: Study 4 of 'Searching the hidden inventory' - can LLM coders show where they are likely wrong? (TalkMoves Subset 2)

Simon Filler
5 links
Zenodo (CERN European Organization for Nuclear Research)
Computational and Text Analysis Methods
article

Preregistration: Study 4 of 'Searching the hidden inventory' - can LLM coders show where they are likely wrong? (TalkMoves Subset 2)

Simon Filler
article en

Abstract

Test of two signals for sentences that LLM coders of teacher talk moves probably get wrong: disagreement among Claude Sonnet 5.5, GPT-6 Luna and Gemini 4 Argon, and the stated confidence of Claude Sonnet 5.5. A frozen annotation harness codes 97 new passages from 8 teachers of TalkMoves Subset 2. Nothing is optimized. The primary measure is Cohen's kappa on the sentences not flagged. Confirmatory hypotheses test each signal against no flagging and compare the two directly, with cluster bootstrap over teachers and transcripts and Holm correction. Registered before the passages were drawn.

Zenodo (CERN European Organization for Nuclear Research)
TU Dortmund University (DE)
Openalex Percentile: Top 7%
Computational and Text Analysis Methods
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon