Assessment of anaesthesia trainees using performance metrics derived from electronic health records: a longitudinal cohort analysis.

Background The development of competencies in anaesthesia residents is assessed by direct observation of their performance and written and oral examinations. Little is known about how residents' competencies are reflected by objective data in anaesthetic records. We investigated whether performance metrics derived from electronic anaesthesia records are associated with anaesthesia residents' progress of training and European written examination timing and results. Methods We recruited 46 anaesthesia providers who trained at the Vienna University Hospital between September 2013 and June 2021, and we reviewed the anaesthesia records of all cases they managed during the study period. We derived six performance measures based on perioperative data routinely collected: intraoperative hypotension and hypothermia, glycaemic control, postoperative nausea and vomiting prevention, lung-protective ventilation, and postoperative kidney injury. We evaluated the association between these quality metrics with training level and written exam completion status in anaesthesia residents after adjusting for patient and surgical factors. Results We found a statistically significant association between the level of training and most performance measures. The probability of preventing intraoperative hypotension increased (OR 1.16, 95% CI 1.12 - 1.20) with the level of training, as did the probability of preventing hypothermia (OR 1.08, 95% CI 1.05 - 1.11) and administering appropriate postoperative nausea and vomiting prophylaxis (OR 1.21, 95% CI 1.15 - 1.27). However, the odds of preventing acute kidney injury decreased (OR 0.91, 95% CI 0.83 - 0.99), as did the use of lung-protective ventilation (OR 0.94, 95% 0.92 - 0.97). All participating residents who took the written exam passed it, precluding a direct pass versus fail comparison, but the exam completion status was associated with increased odds of lung-protective ventilation (OR 1.42, 95% CI 1.03 - 1.95) and decreased odds of preventing intraoperative hypotension (OR 0.7, 95% CI 0.49 - 0.99). Glycaemic control was not associated with either of the training milestones. Conclusions The training level of anaesthesia residents had a significant association with several performance metrics. Passing the written exam only had a modest effect. Performance measures based on patient outcomes and intraoperative care may serve as assessment methods for anaesthesia residents' progress tracking.