Optimising large language models for clinical information extraction: a benchmarking study in the context of ulcerative colitis research

Objective Closed-source large language models (LLMs) like generative pre-trained transformer 4o (GPT-4o) have shown promise for clinical information extraction but are potentially limited by cost, data security concerns and inflexibility. Open-source models are an attractive alternative with various adaptation strategies with no consensus on best practices. This study aims to rigorously identify optimal adaptation strategies for open-source models and evaluate their performance relative to closed-source alternatives. Methods and Analysis We studied three LLM adaptation strategies: chain-of-thought prompting, few-shot prompting and fine-tuning. Our target for information extraction was the Mayo Endoscopic Subscore (MES). We applied those strategies in all combinations to six open-source models (8–70 billion parameters) using an annotated set of colonoscopy procedure reports from the University of California, San Francisco (N=608) and San Francisco General Hospital (N=217). We analysed the relationship of these strategies to several performance metrics with a mixed-effects model, accounting for the variability between centres and LLMs. GPT-4o served as a closed-source oracle and provided in-depth commentary on the cost-effectiveness of these options. Results Quantised low-rank adaptation (QLoRA) statistically improves ( <m:math xmlns:m="http://www.w3.org/1998/Math/MathML" overflow="scroll"> <m:mi>p</m:mi> <m:mo>§amp;lt;</m:mo> <m:mn>0.001</m:mn> <m:mo>)</m:mo> </m:math> ) the performance of open-source LLMs by 9.1–15.7 percentage points across accuracy, precision recall and annotation eligibility accuracy. However, GPT-4o with prompt engineering outperforms the best open-source model by 4.9%–11.2%. A simple cost-effectiveness analysis suggests that GPT-4o is more affordable compared with open-source alternatives. Conclusion GPT-4o is currently the most efficient LLM for MES extraction. If unavailable, QLoRA-optimised open-source models are a competitive alternative. However, results also suggest that current instruction-following LLMs including GPT-4o do not fully follow user-provided instructions, leaving room for improvement. More work is needed to achieve consistent, near-perfect performance in clinical information extraction by LLMs.