Evaluation of Large Language Models for Structured Data Extraction From Interstitial Lung Disease Clinical Notes: Comparative Study

Background Most clinically relevant data are in unstructured clinical notes, which are verbose and imprecise, making structured data extraction a costly bottleneck for screening patients for studies or maintaining health care registries. This challenge is particularly pronounced in interstitial lung disease (ILD) and requires significant human effort to interpret notes and determine classification to create an ILD registry. Large language models (LLMs) have the potential to significantly reduce this cost and effort. Objective We aim to compare the performance of various LLMs for structured data extraction from unstructured ILD clinic notes. Our primary aim was to evaluate LLM extraction of binary structured data (yes/no answers) from clinical notes regarding key ILD clinical questions. A secondary analysis evaluated select LLMs for the extraction of multiclass data to determine ILD classification. Methods We used 12 different LLMs to extract binary answers to 10 ILD clinical questions from the most recent clinic notes of 100 ILD clinic patients. We additionally used 2 LLMs (gpt-oss-20b and gpt-oss-120b) to extract multiclass data regarding ILD classification. Prompts were created with the assistance of ChatGPT (OpenAI) and refined with an iterative approach by testing on a prompt engineering cohort of 10 ILD clinic patient notes. Ground truth was established by consensus among 3 ILD physicians. LLM performance was evaluated using accuracy, precision, recall, and F1-scores. Results LLMs processed each interface call of a clinical note-prompt combination in 1-2 seconds, with estimated costs ranging from less than US $0.001 to US $0.11 (or approximately US $0.05 to US $10.50 per clinical note accounting for 10 runs and 10 binary prompts) depending on the model. Out of the 12 LLMs assessed, 7 models (Claude 3.5 Sonnet [Anthropic], GPT-4o, gpt-oss-20b, gpt-oss-120b, o1, o1-mini, and o3-mini [OpenAI]) performed at human-level accuracy, similar to that of the 3 ILD clinicians (96.2%). A total of 5 LLMs performed significantly worse than humans (Holm-adjusted P≤.003 for all). gpt-oss-120b, o1, and o3-mini models achieved the highest F1-scores of all the evaluated LLMs. There was no significant difference in model accuracy among the top tier models (Claude 3.5 Sonnet, gpt-oss-20b, gpt-oss-120b, o1, o1-mini, and o3-mini), though GPT-4o achieved significantly lower accuracy than o1 (Bonferroni-adjusted P=.04). Multiclass data extraction using gpt-oss-120b and gpt-oss-20b demonstrated lower accuracy when compared to its corresponding binary data extraction (91.1% and 88.0%, respectively). There was no significant difference in accuracy between gpt-oss-120b and gpt-oss-20b for multiclass extraction. Conclusions Multiple LLMs consistently achieved human-level accuracy in extracting structured binary data from ILD clinical notes, while being orders of magnitude faster and cheaper. Multiclass data extraction was possible but associated with a lower accuracy. LLMs are promising tools that can be used for clinical data extraction to improve clinical research efficiency.