Comparison between artificial intelligence-based and manual organ delineations in pretreatment computed tomography scans of prostate cancer patients: a visual grading study

This study aimed to evaluate the clinical acceptability of artificial intelligence (AI)-based organ segmentations on pretreatment CT images of prostate cancer patients using manual organ delineations as a reference. Paired AI-based segmentations and manual delineations of the prostate, urinary bladder, and rectum were evaluated by three observers, according to a 4-grade Likert-scale, based on quality criteria, developed through a Delphi process. Visual grading characteristics (VGC) analysis was performed. When comparing the ratings of AI-based (n = 360) and manual delineations (n = 360), the area under the VGC-curve (AUCVGC) was 0.36 (95% CI 0.27-0.44), 0.35 (95% CI 0.28-0.41), and 0.3 (95% CI 0.22-0.40) for the prostate, urinary bladder, and rectum, respectively, indicating inferior ratings for the algorithm. Few AI segmentations (8%) were considered clinically unacceptable, while in 67% no or minor changes were needed. Despite superior ratings for manual delineations, most AI-segmentations needed no or minor changes, indicating clinical acceptability.