Evaluating the accuracy and reliability of a large language model in coding common orthopaedic procedures

Background: Current Procedural Terminology (CPT) codes are critical for billing and documentation in orthopaedic surgery, yet coding can be complex and costly. Large language models (LLMs) have shown promise in automating CPT coding. Methods: Ten orthopaedic procedures were selected from each of the subspecialties of sports, foot and ankle, and hand. Procedure names were generated by the authors to reflect practice at our institution, and CPT codes for each procedure were secondarily validated for ground truth analysis using the American Association of Professional Coders “Codify” tool. An institutional Health Insurance Portability and Accountability Act (HIPAA)-compliant, ChatGPT-4 Omni (ChatGPT4o)-based LLM was prompted to generate CPT codes based on a procedure name. Three independent trials were performed per procedure. Responses were categorized as “correct”, “partially correct”, or “incorrect” and assigned a score (+1, +0.5, or +0, respectively). Model performance was assessed as a percentage of the maximum possible score. For incorrect responses, the model was prompted to try again. Results: The LLM generated the correct response in 86.7% of trials, with an overall performance of 88.9% of the maximum possible score. Performance varied by subspecialty ( P =0.018), with lower scores in hand compared to sports ( P =0.019). After re-prompting, LLM performance improved to 93.9% of the maximum possible score. The LLM had the most difficulty distinguishing between closely related codes for hand procedures. Conclusions: This study demonstrates promising potential for rapid, accurate, and cost-effective LLM-driven orthopedic coding, suggesting practical applications in reducing administrative workload, improving coding consistency, and potentially lowering costs in orthopaedic clinical practices.