FUO_ED: A Dataset for Evaluating the Performance of Large Language Models in Diagnosing Complex Cases of Fever of Unknown Origin

Fever of Unknown Origin (FUO) has long been a challenging issue in clinical medicine due to the diverse range of possible causes and the difficulty in diagnosis, often leading to misdiagnoses or undiagnosed cases. Recently, the rise of large language models, such as GPT-4, has introduced a new approach to diagnosing FUO. Under specific prompts, GPT-4 has achieved over 90% accuracy on the MedQA dataset, surpassing ordinary medical practitioners. To explore the potential of large language models in addressing such complex clinical cases, we propose the first evaluation dataset for large language models in diagnosing FUO, named the Fever of Unknown Origin Evaluation Dataset (FUO_ED). This dataset is derived from real-world medical records, annotated by large language models and subsequently verified by human experts. It includes 560 questions across 16 representative causes of fever of unknown origin, with each question being evaluated based on descriptions of 180 physiological indicators in patients. We conducted a comprehensive evaluation of the state-of-the-art large language models on FUO_ED. The results show that only GPT-4o achieved an average accuracy of over 40%, indicating that there is significant room for improvement in the application of large language models to real-world FUO diagnostic tasks. We believe FUO_ED will help analyze the strengths and weaknesses of large language models in FUO medical tasks and support their iterative optimization.