Ensuring consistency and fairness: Interrater reliability of the European Respiratory Society thoracic ultrasound Objective Structured Clinical Examination

<bold>Introduction:</bold> The European Respiratory Society thoracic ultrasound training program ends with a final certification, an Objective Structured Clinical Examination (OSCE). To ensure high international standards, several physicians from a variety of countries serve as assessors. However, this might be a disadvantage and cause differences in the assessments. <bold>Aim:</bold> This study aimed to investigate the interrater reliability of the ERS thoracic ultrasound OSCE, providing insight into the consistency and fairness of the certification process. <bold>Methods:</bold> A double-rating process was implemented in selected parts of the 2023 ERS OSCE. One faculty served as the primary assessor, whose evaluation determined the final score, while a second faculty conducted an independent blinded assessment for the purpose of this study. No discussion between the assessors was permitted. Interrater reliability was calculated using Person’s correlation coefficient (r) and internal consistency reliability for each station using Cronbach’s alpha. <bold>Results:</bold> Fifty trainees completed the OSCE. With ten faculty from four different countries serving as assessors, 23 performances were double-rated. The overall interrater reliability across 530 assessment items was high (Pearson’s r=0.75, 95% CI 0.71-0.78). Internal consistency reliability was strong in the three practical stations (Cronbach’s alpha: 0.88 for station 1, 0.92 for station 2, and 0.88 for station 3). The inter-station reliability was moderate (Cronbach’s alpha: 0.59). <bold>Conclusion:</bold> The overall exam demonstrated high internal consistency and good interrater reliability. These findings suggest fairness and robustness despite using various assesors from several countries.