Building an Auditable Reference Standard for Clinical Information Extraction from Transesophageal Echocardiography Reports

Objectives To develop an auditable, two-institution reference standard for intraoperative transesophageal echocardiography (TEE) reports and evaluate whether reviewer disagreement identifies limitations in the underlying clinical information model. Materials and Methods We selected 300 reports (150 per institution) from 925 at the University of Pennsylvania and University of Washington using random sampling and enrichment for challenging documentation. Four trained abstractors applied a shared 39-variable V1 codebook. Independent paired comparisons included 297 reports; three historical reports contributed only to the final reference standard. Categorical agreement used exact agreement and unweighted Cohen's κ; numeric left-ventricular ejection fraction (LVEF) used exact agreement and mean absolute differences. Disagreements underwent structured expert adjudication and reason classification. Additional analyses assessed reviewer-pair, sampling-stratified, and population-weighted agreement. Results Among 11,583 formal reviewer-comparison cells, 10,293 (88.9%) were resolved by reviewer agreement and 1,290 (11.1%) required targeted expert adjudication (1,289 tasks). Codebook/schema ambiguity (40.8%) and interpretation/classification differences (25.2%) together accounted for 66.0% of targeted tasks. Adjudication was concentrated in aortic dissection, valve type, and post-intervention findings. Categorical κ ranged from 0.364 to 0.960 and was inversely correlated with targeted-adjudication burden (Spearman ρ=−0.884; 95% CI, −0.955 to −0.734). Sampling-weighted κ changed by no more than 0.05 for 32 of 35 categorical variables. Discussion Structured adjudication identified recurring schema and interpretation challenges beyond individual abstraction errors, providing specific targets for information-model refinement. Conclusion As NLP and LLM-based abstraction tools move toward clinical use, the reference standards used to evaluate them require comparably rigorous development, adjudication, and independent validation.

Authors

Institutions

Publication Details

Journal
medRxiv
Published
2026-11-20
DOI
https://doi.org/10.64898/2026.11.19.26368500
Primary Topic
Reliability and Agreement in Measurement
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Building an Auditable Reference Standard for Clinical Information Extraction from Transesophageal Echocardiography Reports

Emily J. MacKay, Alison M. Pouch, Annie L. Chen, Rohan Shad et al.
1 links
medRxiv
Reliability and Agreement in Measurement
preprint

Building an Auditable Reference Standard for Clinical Information Extraction from Transesophageal Echocardiography Reports

Emily J. MacKay, Alison M. Pouch, Annie L. Chen, Rohan Shad, Michael S Kaplan, Emily Yu, Layla Canales, Yujia Ge, Mingyang Yuan, Bo Zhang
preprint en

Abstract

Objectives To develop an auditable, two-institution reference standard for intraoperative transesophageal echocardiography (TEE) reports and evaluate whether reviewer disagreement identifies limitations in the underlying clinical information model. Materials and Methods We selected 300 reports (150 per institution) from 925 at the University of Pennsylvania and University of Washington using random sampling and enrichment for challenging documentation. Four trained abstractors applied a shared 39-variable V1 codebook. Independent paired comparisons included 297 reports; three historical reports contributed only to the final reference standard. Categorical agreement used exact agreement and unweighted Cohen's κ; numeric left-ventricular ejection fraction (LVEF) used exact agreement and mean absolute differences. Disagreements underwent structured expert adjudication and reason classification. Additional analyses assessed reviewer-pair, sampling-stratified, and population-weighted agreement. Results Among 11,583 formal reviewer-comparison cells, 10,293 (88.9%) were resolved by reviewer agreement and 1,290 (11.1%) required targeted expert adjudication (1,289 tasks). Codebook/schema ambiguity (40.8%) and interpretation/classification differences (25.2%) together accounted for 66.0% of targeted tasks. Adjudication was concentrated in aortic dissection, valve type, and post-intervention findings. Categorical κ ranged from 0.364 to 0.960 and was inversely correlated with targeted-adjudication burden (Spearman ρ=−0.884; 95% CI, −0.955 to −0.734). Sampling-weighted κ changed by no more than 0.05 for 32 of 35 categorical variables. Discussion Structured adjudication identified recurring schema and interpretation challenges beyond individual abstraction errors, providing specific targets for information-model refinement. Conclusion As NLP and LLM-based abstraction tools move toward clinical use, the reference standards used to evaluate them require comparably rigorous development, adjudication, and independent validation.

medRxiv
Tufts University (US), SUNY Downstate Health Sciences University (US), University of Washington (US), Fred Hutch Cancer Center (US), Villanova University (US), University of Pennsylvania (US), Temple University (US)
Good health and well-being
Reliability and Agreement in Measurement
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon