Development and evaluation of a technology-enhanced simulation to measure physician decision making in trauma triage

Purpose Variable implementation of clinical practice guidelines causes preventable morbidity and mortality. The paucity of valid and reliable measures of physician performance impedes efforts to improve implementation. The objective of this study was to evaluate a low-cost, scalable simulation method to evaluate physician non-adherence to guidelines, using trauma triage in Emergency Departments (EDs) as an archetypal clinical problem. Methods We created an online simulation that mimicked the task environment of a non-trauma center ED. We recruited a sample of ED physicians, asked them to use the simulation, and obtained the electronic health records of injured patients treated by these physicians in the prior 3 years. We used signal detection theory, a behavioral science method, to analyze triage performance. The method quantified the influence of 2 determinants of non-adherence: perceptual sensitivity (diagnostic accuracy) and decisional threshold (revealed preferences for false positive/negative decisions). We collected evidence of the simulation’s response process, internal structure, content validity, and relations with other variables. Results Among 180 invited physicians, 60/180 (33%) enrolled and 45/60 (75%) participated; 38/45 (84%) had accessible electronic records. Physicians completed an average of 20/26 simulation trauma cases, spending 2.6 minutes/case (SD 1.8), and making 2.6 decisions/case (SD 1.7). Responses to similar types of simulation cases were consistent (Cronbach’s alpha 0.80–0.86). Users reported strong content validity. Perceptual sensitivity on the simulation and in real-life correlated moderately well among physicians who evaluated ≥10 severely injured patients/year (rho 0.46, 95% CI 0.07–0.76, p = 0.025), but not lower volumes (rho −0.26, 95% CI −0.87–0.50, p = 0.39); decisional thresholds were uncorrelated (rho 0.22, 95% CI −0.12–0.51, p = 0.18). Conclusions The technology- and behavioral science-enhanced simulation demonstrated evidence of validity across multiple domains, with mixed findings for relations with other variables. With further refinement, it offers a promising avenue for studying physician decision making in trauma triage, with possible application to other clinical domains.