Reliable but Not Interchangeable: What Validated Instruments Measure When They Grade an Anastomosis

Background/Objectives: Multiple validated microsurgical assessment instruments exist that differ in design and theoretical approach. Product-based tools evaluate the quality of the completed anastomosis, while process-based tools assess technique during task execution. Whether these tools measure the same underlying construct and can be used interchangeably has not been directly examined. This study compared the behaviour of three validated instruments, ALI, MARS10, and SMaRT, when applied to identical performances. Methods: Forty-five participants from 26 institutions across 14 countries performed an eight-stitch end-to-end vascular anastomosis on a chicken leg model, independently evaluated by two trained graders using all three instruments. Interrater reliability was assessed using intraclass correlation coefficients (ICC(2,1)), rank-order agreement using Spearman’s rho and Kendall’s W, and score distributions were examined for floor and ceiling effects. Results: All three instruments demonstrated high interrater reliability (ICC: ALI 0.955, MARS10 0.939, SMaRT 0.960), though product-based tools showed wider confidence intervals. Systematic rater differences were observed across all instruments. Rank correlations were strongest between ALI and MARS10 (ρ = 0.913), with moderate correlations between process-based and product-based tools (ρ = 0.698–0.749). MARS10 showed a ceiling effect, SMaRT underutilised its upper range, and ALI had the greatest score variability. Conclusions: Although demonstrating high interrater reliability, the instruments differ in rater variability, discriminatory capacity, and the aspects of performance prioritised, embodying different definitions of surgical skill. Current tools are not interchangeable, and instrument selection carries potential consequences for training and research. Future research in technology-driven approaches could offer a path toward more objective and consistent evaluation that is less dependent on human interpretation.