Peer Review of “Assessing the Limitations of Large Language Models in Clinical Practice Guideline–Concordant Treatment Decision-Making on Real-World Data: Retrospective Study”

for "Assessing the Limitations of Large Language Models in Clinical Practice Guideline-Concordant Treatment Decision-Making on Real-World Data: Retrospective Study." Round 1 ReviewThe authors of this paper [1] set out to determine whether modern large language models (LLMs) can make treatment decisions for severe aortic stenosis based on uncurated, free-text clinical data in routine practice.This question addresses a significant gap in the literature: while earlier work demonstrated that LLMs could agree with expert tumor boards or heart teams when provided with highly structured or preprocessed information, the realities of clinical documentation-discharge summaries, imaging reports, and free-text notes-remain unstructured and noisy.It seems that even top LLMs fail to deliver reliable, unbiased treatment recommendations from raw clinical text.They perform well only with expert-crafted summaries and embedded guidelines, highlighting that data representation (and prompt engineering) is key.