Accuracy of the “Surprise Question” in predicting 90-day mortality among hospitalized patients with decompensated cirrhosis

INTRODUCTION Hospitalized patients with decompensated cirrhosis (DC) are at high risk for short-term mortality.1 The Surprise Question (SQ), that is, “Would you be surprised if this patient died within the next 3 (or 6 or 12) months?”, is a clinician-based prognostication tool initially developed to identify patients who may benefit from palliative care interventions.2 The SQ has been evaluated in several meta-analyses across a range of illnesses (eg, cancer, cardiac, pulmonary, kidney) and clinical settings (eg, inpatient, outpatient, nursing home, hospice) with pooled c-statistic ranging from 0.71 to 0.81, suggesting moderate-to-good prognostic accuracy.2–4 However, this easy-to-use tool has been infrequently studied in DC. The goal of this study was to evaluate the prognostic accuracy of the SQ versus the Model for End-Stage Liver Disease-Sodium score (MELD-Na) for predicting 90-day mortality in a prospective cohort of patients with DC hospitalized at a large academic liver transplant center. METHODS From 2/2022 to 4/2023, patients with DC hospitalized at Massachusetts General Hospital (MGH) were prospectively identified. Eligibility criteria included: (1) ≥18 years old; (2) diagnosis of cirrhosis complicated by ascites, hepatic encephalopathy, and/or variceal hemorrhage; and (3) identified within the first 72 hours of admission to the general medicine service. Patients were excluded if they had an extrahepatic advanced malignancy, were admitted for liver transplantation, or were receiving hospice care. At MGH, patients with DC admitted to inpatient medicine are under the care of the general medicine service, with the hepatology team serving in a consultative role when requested by the primary team. For each identified patient, one treating attending physician and one treating fellow physician, resident physician, or advanced practice provider from internal medicine (IM) and/or hepatology were independently contacted to answer the 90-day SQ (“Would you be surprised if this patient died in the next 90 days?”). Clinicians provided a binary response of “No, I would not be surprised” (positive, SQ+) or “Yes, I would be surprised” (negative, SQ−).5 Clinicians were given 48 hours to complete the 90-day SQ. If none of the treating clinicians responded within the 48-hour window, the patient was excluded from the study. We collected data on patients’ age, gender, race, cirrhosis etiology and complications, MELD-Na score on admission, and transplant listing status from the electronic medical record. Mortality information was ascertained by matching cohort data (first and last names, date of birth) to death records from the U.S. National Death Index and confirmed with medical record review.6 We determined the sensitivity, specificity, positive and negative predictive values (PPV, NPV), and accuracy of the 90-day SQ for predicting 90-day mortality among patients who had responses from: (1) at least 1 IM clinician, (2) at least 1 hepatology clinician, and (3) at least 1 clinician from both IM and hepatology. In patients for whom the 90-day SQ was answered by both clinicians in the same specialty (eg, hepatology attending and fellow), the response of the attending was used for the analysis. We compared the performance of the 90-day SQ versus MELD-Na score on admission using c-statistics, considering a p-value <0.05 (2-sided) as statistically significant.7 All analyses were performed using Python version 3.13.2 and SAS version 9.4. Our protocol was approved by the Mass General Brigham Institutional Review Board. RESULTS In total, 100/122 (82%) eligible patients had clinicians respond within 48 hours of survey administration, of whom 99 had a response from at least 1 IM clinician, 34 from at least 1 hepatology clinician, and 33 from at least 1 clinician from IM and hepatology (Supplemental Figure, https://links.lww.com/HC9/C55). Median patient age was 60 (IQR 49–67), median admission MELD-Na score was 21 (IQR 14–28), and 14% were active on the transplant waitlist (Supplemental Table, https://links.lww.com/HC9/C56). Overall, 25 patients (25%) died within 90 days (Table 1). Among IM clinicians evaluating 99 patients, the 90-day SQ had a PPV of 0.38 (95% CI: 0.25–0.52), NPV of 0.87 (95% CI: 0.76–0.94), and accuracy of 0.65 (95% CI: 0.55–0.73). Among hepatology clinicians evaluating 34 patients, the 90-day SQ had a PPV of 0.44 (95% CI: 0.27–0.63), NPV of 0.78 (95% CI: 0.45–0.94), and accuracy of 0.53 (95% CI: 0.37–0.69). Among the 33 patients who had at least 1 IM and 1 hepatology clinician providing responses, the 90-day SQ had similar accuracy for both clinician groups (0.52; 95% CI: 0.35–0.67). For these 33 patients, there was no significant difference in c-statistics when comparing the MELD-Na score (0.57) versus the 90-day SQ from IM (0.53) or hepatology (0.58) clinicians (p=0.88). TABLE 1 - 90-day Surprise Question test statistics for Internal medicine, Hepatology, and both (N=100) Both (N=33) Internal medicine (N=99) Hepatology (N=34) Internal medicine Hepatology Death at 90 days 24 (24.1) 13 (38.2) 12 (36.4) 12 (36.4) 90-day SQ−a 7 (29.2) 2 (15.4) 5 (41.7) 2 (16.7) 90-day SQ+a 17 (70.8) 11 (84.6) 7 (58.3) 10 (83.3) Sensitivity (95% CI) 0.71 (0.51, 0.85) 0.85 (0.58, 0.96) 0.58 (0.32, 0.81) 0.83 (0.55, 0.95) Specificity (95% CI) 0.63 (0.51, 0.73) 0.33 (0.17, 0.55) 0.48 (0.28, 0.68) 0.33 (0.17, 0.55) PPV (95% CI) 0.38 (0.25, 0.52) 0.44 (0.27, 0.63) 0.39 (0.20, 0.61) 0.42 (0.24, 0.61) NPV (95% CI) 0.87 (0.76, 0.94) 0.78 (0.45, 0.94) 0.67 (0.42, 0.85) 0.78 (0.45, 0.94) Accuracy (95% CI) 0.65 (0.55, 0.73) 0.53 (0.37, 0.69) 0.52 (0.35, 0.67) 0.52 (0.35, 0.67) aDenominator is death at 90 days.Abbreviations: NPV, negative predictive value; PPV, positive predictive value; SQ−, clinician response of “surprised” to the 90-day SQ; SQ, surprise question; SQ+, clinician response of “not surprised” to the 90-day SQ. DISCUSSION In this prospective cohort study of 100 hospitalized patients with DC, of whom 25% had died within 90 days, we found that the 90-day SQ had poor accuracy for predicting 90-day mortality when answered by either IM or hepatology clinicians. Only one prior study has ever evaluated the SQ in hospitalized patients with DC using the 12-month SQ, and the accuracy of the SQ was similarly poor at 0.56.8 Notably, the c-statistics of both the 90-day SQ and the MELD-Na score for 90-day mortality for the 33 inpatients followed by both IM and hepatology clinicians showed both tests performed only slightly better than chance. Our findings highlight the difficulty of predicting short-term mortality for hospitalized patients with DC. Patients with DC have an unpredictable illness trajectory characterized by episodes of sudden decompensation, which can cause rapid decline and death. Traditional prognostic scores such as MELD-Na underestimate the mortality risk of hospitalized patients with DC who develop acute-on-chronic liver failure.9 Similarly, the use of multiple artificial intelligence models has had suboptimal performance for mortality risk prediction in this population.8 Despite these data, our finding that 1 in 4 patients had died in 90 days suggests that all inpatients with DC should be considered high-risk for short-term mortality and may benefit from palliative care interventions.10 Important limitations of our study include that it was conducted at a single academic liver transplant center, a small sample size, and the risk of response bias. Further studies are needed in larger and more heterogeneous clinical samples that include community settings. To conclude, the prognostic accuracy of the 90-day SQ was poor in hospitalized patients with DC, underscoring the difficulty with prognostication in this population.