Standard setting in the era of workplace-based assessment: a pilot exploration

Abstract Purpose The use of workplace-based assessments (WBAs) in competency-based medical education continues to expand rapidly, and there are emerging methods to use these data to establish a trainee’s competence. However, establishing performance standards based on longitudinally collected operative performance data remains difficult, as existing standard-setting in medical education largely focuses on single tasks, single procedures, or written examinations. The purpose of this pilot study was to (1) assess the feasibility of adapting three established standard-setting methods to WBA ratings scored using the Society for Improving Medical Professional Learning (SIMPL) Operative Performance (OP) score and the convergence of the resulting cut scores in defining a general surgery graduation standard; and (2) identify the methodological and process considerations that could inform future WBA-based standard-setting efforts. Methods A panel of 10 surgical educators participated in a two-hour virtual Benchmark Panel. Panelists were oriented to the SIMPL OR WBA tool and the OP score (range 372–628). Panelists applied three standard-setting methods to define a cut score corresponding to a “practice-ready” graduation standard for high-value, high-frequency procedures. Each panelist’s mean OP cut score per method was computed and compared using a one-way repeated-measures analysis of variance (ANOVA) with Holm-adjusted pairwise comparisons. The session transcript was analyzed qualitatively using interpretive description methodology to inductively identify themes to inform future iterations. Results Mean (standard deviation) participant-level OP cut scores were 577 (30) for Angoff, 588 (17) for Bookmark, and 582 (19) for Construct Mapping. There was no association between standard setting method and participant-level OP cut scores [F(2, 18) = 0.63, p = 0.54, generalized η 2 = 0.045]. Holm-adjusted pairwise comparisons were also non-significant (all p > 0.05). Qualitative analysis surfaced themes related to the specific methods, including perceived high utility of Construct Mapping, and overall reflections on the standard-setting process, including the challenge of aligning on the mental model of a “minimally competent graduating trainee” and the difficulty in incorporating variation in case complexity and training experience. A proposed graduation standard of 582, anchored in the Construct Mapping method based on qualitative preference, corresponds to predicted probabilities of earning a practice-ready rating of 88–90% for laparoscopic appendectomy, 83–85% for laparoscopic cholecystectomy, and 76–80% for open partial colectomy. Conclusions Three established standard-setting methods can be feasibly adapted to longitudinal operative performance WBA data and produced participant-level cut scores that were not statistically different. This convergence across methodologically independent approaches supports the proposed OP-score cut of 582 as an empirically grounded starting point for further multi-stakeholder consensus work. Method selection should be guided by methodological trade-offs in resource requirements, data dependencies, and perceived utility. These findings provide an initial framework for setting defensible, data-informed competency standards in the era of WBA.