Clinical Reliability of Large Language Models in Complex Haematology: A Multidimensional Evaluation in Hemophilia–Oncology

Background: The co-existence of hemophilia and cancer presents one of the most complex clinical scenarios, demanding individualised therapeutic planning to balance oncologic efficacy and hemostatic safety. This study evaluated the ability of two Large Language Models (LLMs)—ChatGPT (GPT-4) and Microsoft Copilot (GPT-4–based)—to generate clinically appropriate recommendations for real cases of hemophilia with concurrent malignancy. Methods: Six consecutive adult cases of hemophilia and cancer, managed at the Hemophilia Centre of Padua, Italy, were selected for evaluation. Identical structured prompts were submitted to both LLMs. Two independent expert clinicians rated the model outputs across five domains (Decision/Rationale, Strategy, Selected Drug, Regimen, and Assessment) using a four-level ordinal scale. Results: LLMs demonstrated uneven performances. Outputs were consistently rated as highly reliable in domains involving high-level synthesis, such as Assessment and Strategy. However, substantial limitations were observed in the clinically demanding domains of Selected Drug and Regimen. Critically, in the Selected Drug domain, there was complete agreement between the two expert raters for neither system. This severe lack of concordance signifies that clinicians assigned different adequacy ratings to the same output in every case, reflecting ambiguity, lack of specificity, and inconsistent clinical interpretability of the drug-related information provided by LLMs. Conclusions: While LLMs possess the capacity for high-level reasoning and strategic planning, their inability to translate principles into precise, consistent, and clinically interpretable therapeutic plans—particularly regarding drug selection and treatment regimens—is a significant constraint. These deficiencies, highlighted by the minimal expert concordance in critical domains, necessitate rigorous clinical validation before the responsible integration of LLMs into the management of this uniquely vulnerable patient population.