AKI-BERT: a pre-trained clinical language model for early prediction of acute kidney injury

Acute kidney injury (AKI) is a common clinical syndrome characterized by a sudden episode of kidney failure or kidney damage within a few hours or a few days. Accurate early prediction of AKI for patients in intensive care units (ICUs), who are more likely than others to develop AKI, can enable timely interventions and reduce complications. Much of the clinical information relevant to AKI is captured in unstructured clinical notes, requiring advanced natural language processing (NLP) techniques for effective information extraction. Pre-trained contextual language models, such as Bidirectional Encoder Representations from Transformers (BERT), have recently improved performance across various NLP tasks. This study explores the application of BERT in a disease-specific medical domain task: early prediction of AKI. A domain-specific pre-trained language model, AKI-BERT, was developed by training BERT on clinical notes from patients at risk for AKI. The model’s performance was evaluated using the Medical Information Mart for Intensive Care III (MIMIC-III) dataset. The AKI-BERT model demonstrated improved performance in early prediction of AKI compared to general-domain models. These findings highlight the potential of pre-trained language models tailored to disease-specific medical domains. AKI-BERT expands the utility of the BERT model from general clinical domains to disease-specific applications, demonstrating its effectiveness in mining clinical notes for the early prediction of AKI. This approach has the potential to enhance clinical decision-making and improve patient outcomes in ICUs.