Large Language Models and Responsible AI in Clinical Decision Support Systems: A Systematic Literature Review

The incorporation of large language models (LLMs) into clinical decision support systems (CDSS) offers potential for enhancing the quality of healthcare. Nevertheless, there is a dearth of comprehensive standards and understanding about the application of explainable and responsible AI practices in this context. The objective of this study is to present an overview of the literature on the utilization of LLMs in CDSSs, with an emphasis on Explainable AI (XAI) techniques and responsible AI principles. The present study was conducted through a systematic literature review analyzed of 36 studies on the application of LLMs in CDSS, with a focus on XAI and responsible AI. Human-centered AI is essential to design CDSSs that not only provide accurate recommendations but also align with clinicians’ trust and workflow realities. The results suggest that while LLM-based CDSSs demonstrate potential in enhancing clinical decision-making, concerns regarding model interpretation into healthcare workflows persist as challenges.