Evaluation of Two Commercial Artificial Intelligence Segmentation Systems for Radiation Therapy

Purpose: To assess the clinical acceptability of artificial intelligence (AI) auto-segmentation systems by comparing their performance with physician expert delineated contours, focusing on evaluating contour accuracy and volume differences. Methods: Thirty-three previously treated head and neck patients were selected for comparison. AI-generated structures were created on each computed tomography scan using two AI systems: MIM Protégé AI and TheraPanacea Annotate. Experts participated in a Turing test to distinguish between AI-generated and manually drawn contours. Experts then evaluated the anonymized structure sets on a 3-point scale (1 = lowest acceptance, 3 = highest). Each patient’s structure set was analyzed in MIM, and the volumes (in ml) were recorded. Absolute differences in contour volumes were calculated to assess accuracy. Results: The Turing test showed experienced experts could reliably differentiate between AI-generated and manual contours, while less experienced experts struggled. TheraPanacea outperformed MIM in 60% of the structures based on Dice similarity scores. When TheraPanacea’s Dice scores were lower than MIM’s, the differences were minimal, with the largest gap being 0.049. However, when MIM had lower Dice scores, the differences were more pronounced, with the largest being 0.21 for the Optic Chiasm. Despite TheraPanacea’s higher Dice scores, expert reviews favored MIM; this preference may be due to the long-standing integration of the MIM solution in the clinical workflow, fostering greater familiarity and trust. Conclusions: AI contouring has enhanced the efficiency of radiation therapy planning. This study emphasizes AI-driven auto-segmentation’s ability to streamline organ delineation, despite vendor performance differences.