Reference decisions enhance LLM performance, amplified by source disclosure

Objective The rapid integration of large language models (LLMs) has propelled advancements in automated dialog technologies, improving the public's access to healthcare services. Drawing inspiration from the collaborative decision-making practices of medical professionals in complex cases, we investigated whether LLMs could enhance their diagnostic accuracy through interaction. Methods An experimental study was conducted in China (September–December 2024) to investigate the impact of LLM-generated reference decisions and source disclosure on LLMs’ diagnostic performance. We used a Chinese clinical diagnostic task in a controlled comparative design, where three Chinese LLMs interpreted symptoms and conditions based on patient queries. LLMs’ outcomes were evaluated through accuracy and weighted F1 score metrics, with statistical analysis to determine significance. Results Analysis of variance on LLMs’ diagnostic accuracy scores demonstrated that incorporating LLM-generated decisions as a reference significantly improved diagnostic outcomes, with source disclosure amplifying this improvement. Conclusion Our findings underscore the potential of LLM collaboration in healthcare, offering strategies to refine response generation and decision-making across various applications.