Conformal Prediction and Large Language Models for Medical Coding

Abstract The assignment of current procedure terminology (CPT) codes to medical events is a highly cumbersome, logistic challenge for many healthcare organizations, as well as a significant contributor to medical expenses. Improvement in the allocation of medical resources and expenses dedicated to such tasks can be achieved through automation. However, because of the complex nature of medical records, automation of procedure terminologies is just now developing with the advent of machine learning methods. In this study, we develop a fine-tuned large language model (LLaMA-3B) as a high-reliability predictor for CPT codes. As input, we use 2018 pathology report text data, including gross report, microscopic description, brief medical history, and final diagnosis. We define our dataset with the top five most common technical component CPT codes, which account for 85% of all samples. As a (meta) predictor of the veracity of the large language model itself, we use the prediction’s softmax scalar value of the classification model, borrowing from similar recent approaches in conformal prediction. Specifically, the validation set, which is distinct from both the train and test set, is used to establish a prediction threshold below which the model withholds judgement. We show that, by fine tuning an off-the-shelf language model on pathology report text alone, we can achieve 95% prediction accuracy of the top 5 most common CPT codes on our dataset. We also then demonstrate the utility of combining large language models with conformal prediction. The two in combination raise our accuracy to 99.5% when we allow the model to abstain on making a prediction on 30% of the data, as is determined by a separate threshold on the scalar value of the predictor from the aforementioned validation set. We therefore present a highly flexible model for medical coding, which is simultaneously provably-reliable. Large language models can as such serve as a powerful tool for overcoming challenges in medical billing, thereby improving healthcare efficiency and reducing medical costs.