4CPS-159 Comparative evaluation of ChatGPT-4o and ChatGPT-3.5 in clinical rule-guided dose interventions in hospitalised patients with renal dysfunction

<h3>Background and Importance</h3> Clinical decision support systems (CDSS) play a crucial role in identifying medications that require dose modifications for patients with renal impairment. ChatGPT has the potential to be integrated into electronic health record (EHR) systems for providing such dosing recommendations. Two recent studies, however showed that the performance of ChatGPT-3.5 in this domain was suboptimal.^{1 2} Recently, ChatGPT-4o was released, offering more contextually relevant responses compared to previous versions. In this study, we want to compare the performance of ChatGPT-4o with version 3.5 in clinical rule-guided dose interventions for hospitalised patients with renal impairment. <h3>Aim and Objectives</h3> To evaluate the effectiveness of ChatGPT-4o compared to ChatGPT-3.5 in providing clinical rule-guided dose interventions for patients with renal impairment. <h3>Material and Methods</h3> This cross-sectional study was conducted at a teaching hospital in Europe. CDSS alerts related to renal dysfunction were collected from the EHR over a two-week period and presented to both ChatGPT versions and an expert panel. Alerts were presented with and without patient-specific variables, and suggested medication interventions were compared to assess performance. <h3>Results</h3> A total of 172 CDSS alerts were generated for 80 patients. For alerts without patient variables, ChatGPT-4o provided ‘correct and identical’ responses to 41.3% of the alerts, ‘correct but different’ responses to 18.6%, and ‘incorrect’ responses to 40.1%. In contrast, ChatGPT-3.5 provided ‘correct and identical’ responses for only 19.9% of the alerts, ‘correct but different’ responses for 26.7%, and ‘incorrect’ responses for 53.4%. When alerts included patient-specific variables, ChatGPT-4o offered ‘correct and identical’ responses in 39.0% of cases, ‘correct but different’ responses in 12.2%, and ‘incorrect’ responses in 48.8%. In comparison, ChatGPT-3.5 provided ‘correct and identical’ responses for only 16.7%, ‘correct but different’ for 16.0%, and ‘incorrect’ responses for 67.3% of the alerts. Notably, both versions of ChatGPT showed better accuracy for newer drugs, such as direct oral anticoagulants. <h3>Conclusion and Relevance</h3> ChatGPT-4o demonstrated significantly better performance compared to ChatGPT-3.5 in clinical rule-guided dose interventions for patients with renal dysfunction, showing an improvement in accuracy and a reduction in incorrect recommendations. However, despite these improvements, ChatGPT-4o is insufficient for automatic integration into EHR systems to manage CDSS alerts for renal dysfunction. <h3>References and/or Acknowledgements</h3> https://ejhp.bmj.com/content/early/2023/05/31/ejhpharm-2023-003750.long https://link.springer.com/article/10.1007/s00228-024-03687-5 <h3>Conflict of Interest</h3> No conflict of interest