Identifying Patient Preferences for Information About Healthcare AI: A Discrete Choice Experiment

workflow.However, LLMs sometimes ignores important contextual information, generating inaccurate responses.Moreover, the generated response may be overly general, lack transparency in clinical reasoning, and may not align with clinical practice guidelines.These drawbacks currently limit widespread clinical use.Objectives: To seamlessly integrate clinical practice guidelines into LLMs and enhance their clinical decision support capabilities.Methods: We focused on supporting outpatient treatment decisions for COVID-19.Because keeping up with the frequent COVID-19 guideline updates was difficult for healthcare providers.We used the CDC and Infectious Diseases Society of America (IDSA) COVID-19 Outpatient Treatment Guidelines and utilized in-context learning to incorporate clinical practice guidelines and enhance LLMs' decision-support capabilities.The process of in-context learning consists of two steps: 1) a system prompt that integrates the COVID-19 Outpatient Treatment Guidelines, which was knowledge-engineered as a binary decision tree, and 2) few-shot learning prompts that provide five examples (5-shot) illustrating the treatment recommendation process.We implemented GPT-3.5 and GPT-4 through OpenAI API (temperature 1) and compared them with the LLaMA and Google's PaLM.We denote the proposed ChatGPT with clinical guidelines as ChatGPT-CARE (GPT-3.5)and ChatGPT-CARE (GPT-4) and the LLaMA and PaLM as LLaMA-CARE and PaLM-CARE, respectively.To evaluate the LLMs, we developed 45 snippets of various synthetic COVID-19 patients.We compared LLMs' treatment recommendation results with physicians' gold standard in terms of precision, recall, and F1 score.Results: Quantitative evaluation results are shown in Table 1.ChatGPT-CARE (GPT-4) achieved the best performance of 1.00 in terms of F1 score. Conclusion:The results demonstrate ChatGPT-CARE's higher accuracy compared to ChatGPT and other LLMs in across a range of cases, showcasing its improved inference capabilities.The proposed LLMs can be easily applied to other clinical conditions.