EP24.30: Pilot study of intra‐ and interoperator agreement of a proposed image quality scoring system for transvaginal ultrasound images

This study aimed to test the intra- and interoperator reliability of a proposed scoring system to quantify TVUS image quality, which could be used in the development of artificial intelligence(AI) tools for diagnosis of endometriosis. A proposed three-point scoring system was developed from review of existing literature and expert assessment. A core of 1-3 was assigned, where 1 = image quality is poor, 2 = image quality is suboptimal and 3= image quality is optimal. If the image was deemed to be inaccurate, it was assigned a score of 0 and rejected. Six raters (two sonologists, two sonographers and two radiologists) assigned 150 images (50 cases [50 uterus and 100 ovaries]) a score of 0-3. This was repeated after a period of at least one week. Interobserver reliability was calculated with an Intraclass correlation coefficient (ICC) and intraobserver reliability was calculated with a weighted kappa (k). Poor levels of interobserver agreement were obtained between all six raters for all 150 images (ICC = .480), and for uterus only (ICC = .359). Moderate levels of agreement were achieved for images of the ovaries (ICC = .531). Agreement between the paired sonologists and sonographers was poor for all images (ICC = .336 and .425) as well as uterus (ICC = .253 and .299) and ovaries (ICC= .400 and .469) only. Moderate levels of agreement were achieved between the paired radiologists overall (ICC = .600) and for the uterus (ICC = .538) and ovaries (ICC = .621). Intraobserver agreement was weak to moderate among each of the raters overall (range k = 0.533-0.718, ICC = 0.636-0.819) and for the ovaries (range k = 0.467-0.751, ICC = 0.596-0.862). Weak to strong agreement was seen for the uterus (range k = 0.568-0.808, ICC = 0.546-0.983). All measures were statistically significant (p < .001). The proposed scoring system produced poor-moderate interobserver reliability and mostly weak to moderate levels of intraobserver reliability. More refinement of the scoring system may be needed to improve reliability.