In search of truth: evaluating concordance of AI-based anatomy segmentation models

Purpose: Artificial intelligence based methods for anatomy segmentation can help automate characterization of large imaging datasets. The growing number of similar functionality models raises the challenge of evaluating them on datasets that do not contain ground truth annotations. We introduce a practical framework to assist in this task. Approach: We harmonize the segmentation results into a standard, interoperable representation, which enables consistent, terminology-based labeling of the structures. We extend 3D Slicer to streamline loading and comparison of these harmonized segmentations and demonstrate how standard representation simplifies review of the results using interactive summary plots and browser-based visualization using the OHIF Viewer. To demonstrate the utility of the approach, we apply it to evaluating segmentation of 31 anatomical structures (lungs, vertebrae, ribs, and heart) by 6 open-source models-TotalSegmentator 1.5 and 2.6, Auto3DSeg, MOOSE, MultiTalent, and CADS-for a sample of computed tomography scans from the publicly available National Lung Screening Trial dataset. Results: We demonstrate the utility of the framework in enabling automating loading, structure-wise inspection, and comparison across models. Preliminary results ascertain the practical utility of the approach in allowing quick detection and review of problematic results. The comparison shows excellent agreement segmenting some (e.g., lung) but not all structures (e.g., some models produce invalid vertebrae or rib segmentations). Conclusions: The open-source resources developed include segmentation harmonization scripts, interactive summary plots, and visualization tools. These resources assist in segmentation model evaluation in the absence of ground truth, ultimately enabling informed model selection.