A comprehensive evaluation of large language models for information extraction from unstructured electronic health records in residential aged care

Despite rapid healthcare digitization, extracting information from unstructured electronic health records (EHRs), such as nursing notes, remains challenging due to inconsistencies and ambiguities in clinical documentation. Generative large language models (LLMs) have emerged as promising tools for automating information extraction (IE); however, their application in real-world clinical settings, such as residential aged care (RAC), is limited by critical gaps. Prior studies have often focused on structured EHR data and conventional evaluation metrics such as accuracy and F1 score, overlooking critical aspects like robustness, fairness, bias, and contextual relevance, particularly in unstructured clinical narratives. To address these gaps, this study develops a holistic evaluation framework for clinical IE from free-text nursing notes in the Australian RAC. We systematically evaluate 17 LLMs, including general-purpose and healthcare-specific variants (e.g., LLaMA, Mistral, Gemini, T5) across retrieval-augmented generation (RAG) frameworks and few-shot learning configurations (one-shot, three-shot, four-shot, five-shot). The evaluation focuses on two clinical IE tasks: named entity recognition (NER) and summarization. Results reveal LLaMA 3.1 achieved 88.58 % accuracy, 87.43 % F1 score in NER, 88.18 % F1 score, and 83.15 % relevance in summarization. However, robustness remained low (4.00 % for NER, 4.31 % for summarization) despite excellent fairness (99.9 %) and minimal bias (0.11 %) in both tasks. Further, healthcare-specific LLMs slightly outperform general models, and RAG-based approaches (LangChain, LlamaIndex) yield superior results. Task-specific optimal few-shot settings emerged: three-shot for NER and five-shot for summarization. This study provides a foundation for safely integrating generative AI into clinical decision support.