Leveraging Prompt Engineering with Lightweight Large Language Models to Label and Extract Clinical Information from Radiology Reports

Chest X-ray imaging plays a critical role in diagnosing chest diseases, making it a cornerstone in clinical and research domains. Automating disease diagnosis and extracting relevant clinical information from chest X-ray reports have become essential for developing AI-driven healthcare systems. While effective, deep learning models require extensive labelled datasets, making the labelling of diseases from radiology reports crucial. Traditionally, rule-based labelling approaches have been employed, but the emergence of large language models (LLMs) has introduced new possibilities through instruction-based prompt engineering. In this study, we explore various prompt engineering techniques, including in-context learning and prompt chaining, to label multilabel disease reports and extract key clinical findings from radiology reports. We conducted ablation studies on both proprietary LLMs (e.g., GPT-4 Turbo, GPT-3.5 Turbo) and publicly available LLMs (e.g., Llama2-7B, Llama2-13B, Llama3-8B, Llama2-70B), comparing their performance in terms of clinical accuracy, privacy, and computational cost. Our findings demonstrate that well-crafted prompts on publicly available and lightweight LLMs can achieve competitive results compared to larger and/or proprietary models, offering a cost-effective and privacy-preserving solution for clinical applications. These results highlight the potential of leveraging advanced prompt engineering to streamline disease labelling and enhance the quality of automated report generation in radiology.