Preliminary evaluation of a typed decision model for counselling dialogue coding using motivational interviewing
Typed decision models return probabilities over predefined options, potentially supporting automated coding of motivational interviewing transcripts. The study compared Jev 1.13 with ten experts and two large language models (LLMs), Claude Opus 5.5 and GPT-6 sol, on seven motivational interviewing demonstrations. LLMs reported confidence as probabilities. Jev showed high panel agreement, although Claude Opus 5.5 had higher point estimates on both tasks. Against the ten-expert plurality, its estimated calibration error was lower than both LLMs', clearly so only against Claude Opus 5.5. Calibration comparisons depended on the reference; a prespecified target was missed against single coders. Retained successful LLM calls cost 64–175 times as much as Jev’s, with median response times 7–8 times as long. Costs and latencies were measured through one provider on one day. These preliminary findings motivate evaluation in independent clinical sessions and workflows combining automated coding with human review.
Authors
- Mor Bar (ORCID: https://orcid.org/0000-0002-8712-0309)
- Zohar Elyoseph (ORCID: https://orcid.org/0000-0002-5717-4074)
- Elad Refoua (ORCID: https://orcid.org/0000-0003-1471-3573)
- David Piterman (ORCID: https://orcid.org/0000-0002-2649-8359)
- Hadar Dery (ORCID: https://orcid.org/0009-0006-8341-0743)
- Ziv Ben-Zion
- Shaked Seker
Institutions
- Harvard University (US)
- Bar-Ilan University (IL)
- Universität Trier (DE)
- University of Haifa (IL)
Publication Details
- Journal
- Research Square
- Published
- 2026-11-20
- DOI
- https://doi.org/10.21203/rs.3.rs-11596861/v1
- Primary Topic
- Psychotherapy Techniques and Applications
- Type
- preprint
Funders
- Azrieli Foundation