Large-Scale Deployment of Large Language Models to Extract Hemodynamic Variables from Cardiac Catheterization

Background: Pulmonary hypertension (PH) is a morbid but underrecognized disease diagnosed by right heart catheterization (RHC). Automated extraction of RHC data from electronic health systems is likely to expedite PH diagnosis. However, RHC outputs are typically unstructured, free-text reports and current large language models (LLM) that extract patient-level RHC data are imprecise and derived from small datasets. Methods and Results: We established an iterative development and validation framework using the LLM GPT-4-0125-preview to extract patient-level data from RHC reports. The extraction accuracy was assessed against manual expert-curated ground truth using micro-averaged precision, recall, and F1 metrics across all extracted variables. Evaluation on an independent test set of 100 previously unseen reports yielded a precision of 99.8%, a recall of 99.5%, an F1 score of 99.8%, and a hallucination rate of 0.09%. Next, we deployed the LLM to all n = 17,151 RHC reports captured in the Electronic Health Record of the University of Maryland Medical System (UMMS; ~2.3 million patients across 12 hospitals) from 1 January 2016 to 31 December 2024. Manual validation on 200 random reports demonstrated a precision of 99.7%, a recall of 99.2%, an F1 score of 99.5%, and a hallucination rate of 0.17% (all realistic and physiologically plausible). To further explore reproducibility across deployment settings, we deployed the LLM to a targeted sample of 3000 randomly selected RHC reports that included previously identified error cases. Validation of 200 random reports demonstrated a hallucination rate of 0.33% (precision 99.4%, recall 99.9%, F1 99.6%). Notably, four of the seven observed hallucinations in the large-scale deployment were absent when processed in a smaller deployment cohort. Conclusions: We demonstrate high precision for LLMs when deployed on a scale to RHC reports from a large, diverse multi-hospital EHR. Hallucinations were uncommon overall, but occasional report-level extraction errors persisted despite prompt refinement, highlighting the importance of continued validation during large-scale deployment.

Authors

Institutions

Publication Details

Journal
BioMedInformatics
Published
2026-11-19
DOI
https://doi.org/10.3390/biomedinformatics6060111
Primary Topic
Machine Learning in Healthcare
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Large-Scale Deployment of Large Language Models to Extract Hemodynamic Variables from Cardiac Catheterization

Katarina Zeder, Devina Chatterjee, Bradley A. Maron, Mary Elizabeth Maldarelli et al.
BioMedInformatics
Machine Learning in Healthcare
article

Large-Scale Deployment of Large Language Models to Extract Hemodynamic Variables from Cardiac Catheterization

Katarina Zeder, Devina Chatterjee, Bradley A. Maron, Mary Elizabeth Maldarelli, Colleen M. Ennett, Parvin Mohammadiarvejeh, Yijun Yin, Clayton Brown
article en

Abstract

Background: Pulmonary hypertension (PH) is a morbid but underrecognized disease diagnosed by right heart catheterization (RHC). Automated extraction of RHC data from electronic health systems is likely to expedite PH diagnosis. However, RHC outputs are typically unstructured, free-text reports and current large language models (LLM) that extract patient-level RHC data are imprecise and derived from small datasets. Methods and Results: We established an iterative development and validation framework using the LLM GPT-4-0125-preview to extract patient-level data from RHC reports. The extraction accuracy was assessed against manual expert-curated ground truth using micro-averaged precision, recall, and F1 metrics across all extracted variables. Evaluation on an independent test set of 100 previously unseen reports yielded a precision of 99.8%, a recall of 99.5%, an F1 score of 99.8%, and a hallucination rate of 0.09%. Next, we deployed the LLM to all n = 17,151 RHC reports captured in the Electronic Health Record of the University of Maryland Medical System (UMMS; ~2.3 million patients across 12 hospitals) from 1 January 2016 to 31 December 2024. Manual validation on 200 random reports demonstrated a precision of 99.7%, a recall of 99.2%, an F1 score of 99.5%, and a hallucination rate of 0.17% (all realistic and physiologically plausible). To further explore reproducibility across deployment settings, we deployed the LLM to a targeted sample of 3000 randomly selected RHC reports that included previously identified error cases. Validation of 200 random reports demonstrated a hallucination rate of 0.33% (precision 99.4%, recall 99.9%, F1 99.6%). Notably, four of the seven observed hallucinations in the large-scale deployment were absent when processed in a smaller deployment cohort. Conclusions: We demonstrate high precision for LLMs when deployed on a scale to RHC reports from a large, diverse multi-hospital EHR. Hallucinations were uncommon overall, but occasional report-level extraction errors persisted despite prompt refinement, highlighting the importance of continued validation during large-scale deployment.

BioMedInformaticsVol. 6(6)
University of Maryland, Baltimore (US), University of Maryland Medical System (US)
Machine Learning in Healthcare
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon