Validation of a Behavioral Observation System for Evaluating Routine Gonadectomy Surgical Skill

Abstract Assessment of surgical skills for educational research purposes is challenging. Behavioral observation systems (BOSs) seek to quantify specific behaviors by observation. The purpose of this study was to develop and provide evidence supporting the validity of a BOS for evaluating surgical skills during gonadectomy. A BOS was developed with input from four American College of Veterinary Surgery board-certified surgeons through a Delphi process. Twelve gonadectomy procedures were filmed, six each from veterinary students and board-certified surgeons. The BOS was designed to count the frequency of skills observed during key steps of a gonadectomy procedure. Four evaluators performed scoring twice, a minimum of 3 months apart. The BOS scores were compared between the surgeons and students using a multivariate linear regression with BOS as the dependent variable and the evaluation time point, evaluator, and surgeon type as independent variables. Intra-and interrater reliability were calculated using the intraclass correlation coefficient (ICC). Significance was set at α = 0.05. An initial list of 33 surgical skills was reduced to a BOS with 11 skills. There was a significant difference in the BOS between student surgeons and board-certified surgeons (–13; 95% CI = –15.5 to –10.6; p < .001). The interrater ICC was 0.814 (0.690–0.904), and the intrarater ICC was 0.890 (0.812–0.937). There was a significant relationship between BOS and duration of surgery time ( p = .03, R2 = 0.38). A behavioral observation system was developed and evidence provided for content validity, response processes, reliability, and relationships with other variables. Using the BOS described here in future research pursuits will continue to accrue evidence for validity and produce data that are valid and reliable. This will enhance the quality of the data used to make decisions about student surgical education.