Harnessing Large Language Models: Advancing ADHD Care Quality, Fairness, and Transparency With AI

In this issue of Pediatrics, Bannett and colleagues present a compelling study demonstrating the potential of large language models (LLMs) to enhance quality-of-care assessments, specifically in the management of attention-deficit/hyperactivity disorder (ADHD).1 Applying an open-source LLM to clinical notes from a community-based pediatric network provides a scalable solution for monitoring clinician adherence to guidelines on side effects management following ADHD medication prescriptions. This work advances our understanding of the role of artificial intelligence (AI) in automating and scaling clinical practice assessments, overcoming traditional limitations such as the labor-intensive nature of chart reviews. We appreciate the ample analytic details provided by the authors, which improves the transparency of their work, facilitates the application of their validated methods in new settings, and supports adaptation of their methods to new clinical problems.The authors’ work is important: the failure of clinical teams to monitor and act on ADHD medication side effects may result in discontinuation of treatments, loss of health care system trust, and school failure.2,3 Their analyses revealed 3-fold variability in rates of side effects inquiry across pediatric practices (range 23%–78% of encounters) and even greater variability among providers within practices. Additionally, they were able to draw on their analysis of more than 15 000 clinical notes for children aged 6 to 11 years with ADHD to identify targeted areas for practice improvement. Specifically, they observed that 7 of their 11 participating practices relied on telehealth for more than half of their ADHD follow-up, with side effects inquiry documented in only half (51.9%) of those interactions. The gap in addressing medication side effects is a significant opportunity to improve the quality of care, especially by helping clinical teams remember to address side effects during telehealth encounters. For example, during the 90-day period after initiating or a changing a child’s ADHD treatment, the authors’ algorithm could monitor progress notes for discussions of side effects and prompt clinicians to capture such information if it has not already been recorded. These prompts could include alerts at the time of a follow-up visit or well-timed messages between visits suggesting a follow-up telehealth visit if no follow-up visits have occurred.The LLM algorithm developed by the authors is highly accurate and likely more consistent than typical chart audit methods.4,5 In their deployment sample, the LLM model achieved high sensitivity (88.0%) and specificity (90.1%) in identifying notes documenting side effects inquiries.Any use of new technologies such as AI in health care must consider the possibility that the technology might further exacerbate such disparities. This is particularly important in the context of LLMs, which are trained on a vast corpus of documents that embed and reflect biases that may be rampant in some of the publicly available sources of training data for these complex models. This study includes a fairness analysis, finding no significant bias in algorithm accuracy in studied patient subgroups among whom social drivers of health may affect (1) the quality of care received, (2) the way clinicians document care that has been provided, and/or (3) highly relevant outcomes such as school performance. However, the authors’ use of insurance type as a proxy for socioeconomic status (SES) is known to be inaccurate, misassigning SES in up to 23% of individuals in a recent study.6 The use of area deprivation index (neighborhood-level deprivation) would have provided additional reassurance and should be considered in future assessments of fairness.7The study’s implications extend beyond ADHD care. It highlights the broader potential of LLMs to enhance health care quality by providing real-time insights into clinician adherence to practice guidelines. These models could serve as tools for continuous improvement, offering feedback loops to clinicians and health organizations. By identifying gaps in practice, such as the underdocumentation of side effects in telephone or telehealth encounters or in patients prescribed nonstimulant medications, LLMs present actionable opportunities for health care providers to refine care and improve patient outcomes.In conclusion, the deployment of LLMs for quality-of-care assessments represents a promising step toward more efficient, scalable, and equitable health care practices. The findings from this study underscore the role of AI in fostering a learning health system where clinical care is continuously monitored and enhanced in real time, ultimately benefiting children with ADHD and beyond.