Evaluating healthcare quality and inequities using generative AI: a simulation study of potentially inappropriate medication among older adults analyzed via the framework analysis of individual heterogeneity and discriminatory accuracy (AIHDA)

Abstract Background Regular monitoring of healthcare quality and equity is crucial for informing decision-makers and clinicians. This study explores the application of generative AI, more specifically large language models (LLMs), to facilitate standardized monitoring of healthcare quality using the established framework Analysis of Individual Heterogeneity and Discriminatory Accuracy (AIHDA). The study investigates whether a customized GPT can effectively apply the AIHDA-framework to assess healthcare quality in a simulated dataset. Population and methods Using simulated data modelled on real-world healthcare information, we evaluated the quality indicator of potentially inappropriate medication (PIM). A customized GPT built on ChatGPT 4o was prompted via the principle TREF (Task, Requirement, Expectation, Format) to perform the analysis. Results were compared to a traditional analysis performed with Stata to evaluate accuracy and reliability. Results The GPT successfully conducted the AIHDA analysis, producing results equal to those of the Stata analysis. The GPT provides useful visualizations and structured reports as well as interactive dialog with the end-user in real-time. However, occasional variations in the results occurred in some iterations of the analysis, highlighting potential issues with reliability. The analysis requires close supervision, as the GPT presents both errors and correct results with confidence. Conclusions Generative AI and LLMs show promise in supporting standardized monitoring of healthcare quality and equity using the AIHDA-framework. It enables accessible analysis but requires oversight to address limitations such as occasional inaccuracies. Future and more reliable models of LLMs and local deployment on secure servers may further enhance the utility for routine healthcare monitoring.