Validation of the Opioid Risk Tool for predicting opioid use disorder in an emergency department population

The United States National Institute on Drug Abuse recommends the Opioid Risk Tool (ORT) as a prognostic tool for opioid use disorder (OUD) risk stratification among patients with pain, but performance may vary over time and across populations or settings. Continuous validation is critical to avoid misguided clinical decisions. Prior validation studies were limited to assessing discrimination, which alone provides incomplete assessment of performance. We aimed to evaluate performance of the ORT for predicting OUD in a contemporary cohort with a focus on calibration and clinical utility. We used electronic health record data from an urban health system to identify patients aged ≥18 years, administered the ORT during an emergency department encounter between January 2021 and March 2023, and prescribed an opioid for pain 30 days before or after their index ORT. Our outcome of interest was incident OUD within 365 days. We assessed ORT performance using the area under the receiver operating characteristic curve (AUC), calibration measures, and decision curve analysis to quantify clinical utility. Our study population comprised 21,266 patients. The median age was 48 years, 51% were male, 65% were racial/ethnic minorities, 12% had a history of serious mental illness, and 1.2% (n=244) were diagnosed with OUD during follow-up. The ORT had an AUC of 0.65 (95% confidence limits [CL]: 0.61, 0.69) with severe overestimation of risk, and too extreme risk estimates. The ORT had net benefit at risk thresholds below 3.5% but net harm above this threshold. The ORT had moderate discrimination, but calibration was poor. Clinical utility of the ORT is limited to an extremely cautious strategy for implementing precautionary measures and incompatible with the originally recommended threshold. Further studies should update model predictions to correct miscalibration and expand on comprehensive assessment of performance across populations and settings.