Preregistration: Study 4 of 'Searching the hidden inventory' - can LLM coders show where they are likely wrong? (TalkMoves Subset 2)
Test of two signals for sentences that LLM coders of teacher talk moves probably get wrong: disagreement among Claude Sonnet 5.5, GPT-6 Luna and Gemini 4 Argon, and the stated confidence of Claude Sonnet 5.5. A frozen annotation harness codes 97 new passages from 8 teachers of TalkMoves Subset 2. Nothing is optimized. The primary measure is Cohen's kappa on the sentences not flagged. Confirmatory hypotheses test each signal against no flagging and compare the two directly, with cluster bootstrap over teachers and transcripts and Holm correction. Registered before the passages were drawn.
Authors
- Simon Filler (ORCID: https://orcid.org/0009-0008-8736-8831)
Institutions
- TU Dortmund University (DE)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-02
- DOI
- https://doi.org/10.5281/zenodo.23896103
- Primary Topic
- Computational and Text Analysis Methods
- Type
- article
- Field-Weighted Citation Impact
- 0.00