Author's Response to: On Evaluating <scp>GPT</scp> ‐4 for <scp>CHB</scp> Treatment Recommendations: Reproducibility, Vignette Design and Prompting

We thank Sun and colleagues [1] for their thoughtful comments on our study of guideline coherence of LLM recommendations in chronic hepatitis B. We evaluated a single LLM response per case and condition (with and without context) to reflect the most likely clinical use case in which physicians typically prompt the LLM once rather than averaging across multiple queries. Besides, contextualization showed similar improvements for all five cases, reducing the likelihood of a chance finding. However, we agree that future work should assess the generalizability of the results by including multiple prompts per case to capture LLM response heterogeneity [2]. The selection of case vignettes in our study was highly constrained by the changes between the old and new WHO guidelines. While this design choice was essential to expose limited LLM training on recent content changes, it naturally limits the generalizability of our findings. Moreover, quantifying case difficulty is inherently challenging [3] and further complicated by variability in users' clinical expertise. Expanding both the number and topical range of cases would allow a more comprehensive assessment of the impact of contextualization in the future. While we acknowledge that different prompting strategies influence LLM outputs, we focused on the simplest, most accessible approach for LLM queries: attachment of the full guideline PDF as it reflects how most physicians are likely to engage with LLMs. Pre-processing of guidelines in terms of reformatting or utilisation of more advanced approaches like retrieval augmented generation pipelines may improve response accuracy of LLMs even further as indicated by recent research [4]. These enhancements would complement our findings but are unlikely to alter the central conclusion: LLMs require access to the latest external sources to provide correct clinical advice. Our primary evaluation relied on clinician-based ratings with reported inter-rater agreement, mitigating subjectivity by averaging ratings across multiple reviewers and applying predefined categories. However, we agree that objective text-based metrics would complement expert review and help standardise ‘clarity’ and ‘completeness’ across studies. The authors declare no conflicts of interest. This article is linked to https://doi.org/10.1111/liv.70375. The authors have nothing to report.