Artificial Intelligence for automatic and objective assessment of competencies in flexible bronchoscopy.

<bold>Introduction:</bold> Bronchoscopy is a challenging technical procedure, and assessment of competence currently relies on expert raters. Human rating is time consuming and prone to rater bias. <bold>Study Aim:</bold> To evaluate if a bronchial segment identification system based on Artificial Intelligence (AI) could automatically, instantly, and objectively assess competencies in flexible bronchoscopy in a valid way. <bold>Methods:</bold> Participants were recruited at the Clinical Skills Zone of the European Respiratory Society Annual Conference in Milan, 9-13th September 2023. The participants performed one full diagnostic bronchoscopy in a simulated setting and were rated immediately by AIBA according to its four outcome measures: Diagnostic Completeness (DC), Structured Progress (SP), Procedure Time (PT), and Mean Intersegmental Time (MIT). The procedures were video-recorded and rated after the conference by two blinded, expert raters using a previously validated assessment tool with nine items regarding anatomy and dexterity. <bold>Results:</bold> Fifty-two participants from six different continents were included. All four outcome measures of AIBA correlated significantly with the experts’ anatomy-ratings (Pearson’s correlation coefficient, P-value): DC (.47, P<.001), SP (.57, P<.001), PT (−.32, P=.021), and MIT (−.55, P<.001) and also with the experts’ dexterity-ratings: DC (.38, P=.006), SP (.53, <.001), PT (−.34, P=.014), and MIT (−.47, P<.001). <bold>Conclusions:</bold> The study provides initial validity evidence for AI-based immediate and automatic assessment of anatomical and navigational competencies in flexible bronchoscopy. SP provided stronger correlations with human experts' ratings tha the traditional DC.