Beyond benchmarks: Towards robust artificial intelligence bone segmentation in socio-technical systems

Despite the advances in automated medical image segmentation, AI models still underperform in various clinical settings, posing challenges for integration into real-world workflows. In this pre-registered prospective multicenter evaluation, we analyzed 20 state-of-the-art mandibular segmentation models across 19,218 segmentations of 1,000 clinically resampled CT/CBCT scans. Our results suggest that for a given model, segmentation accuracy can vary by up to 25% in Dice score as socio-technical factors such as voxel size, bone orientation, and patient conditions (e.g., osteosynthesis or pathology) shift from favorable to adverse. Higher sharpness, isotropic smaller voxels, and neutral orientation significantly improved results, while metallic osteosynthesis and anatomical complexity led to significant degradation. Our findings challenge the common view of AI models as “plug-and-play” tools and suggest evidence-based optimization recommendations for both clinicians and developers. This will in turn boost the integration of AI segmentation tools in routine healthcare.

Authors

Institutions

Publication Details

Journal
Expert Systems with Applications
Published
2025-10-17
DOI
https://doi.org/10.1016/j.eswa.2025.130031
Citations
3
Primary Topic
Radiomics and Machine Learning in Medical Imaging
Type
article
Field-Weighted Citation Impact
4.08
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Beyond benchmarks: Towards robust artificial intelligence bone segmentation in socio-technical systems

Pieter-Jan Verhelst, Maxime Gillot, Dennis Madsen, Tobias Pankert et al.
3 citations
Expert Systems with Applications
Radiomics and Machine Learning in Medical Imaging
4.08
article

Beyond benchmarks: Towards robust artificial intelligence bone segmentation in socio-technical systems

Pieter-Jan Verhelst, Maxime Gillot, Dennis Madsen, Tobias Pankert, Kunpeng Xie, Jan Egger, Zhibin Xu, Negar Shahamiri, Victor Alves, Jan Schepers, Daniel Truhn, Oldřich Kodym, Rainer Röhrig, Gauthier Dot, Jonas Bienzeisler, Behrus Puladi, Stefaan Bergé, Lennart Johannes Gruber, Michel Beyer, André Ferreira, Ashkan Rashad, Jiangchang Xu, Adriaan Lambrechts, Stefan Raith, Elias Tappeiner, Osku Sundqvist, Guomin Wu, Robert Radu Ileșan, Lucía Cevidanes, Alexander Hermans, Frank Hölzle, Tong Xi, Yao Li, Florian M. Thieringer, Tabea Flügge, Hans Lamecker, Niels van Nistelrooij, Joep Kraeima, Constantin Seibold, Max J. H. Witjes, Stefan Zachow, Shankeeth Vinayahalingam, Sven Nebelung, Xiaojun Chen, Yan Guo, A. Coppens, Ross Cotton, Reinier ten Brink, Charlotte Weschke, Martin Crampen, Thomas Potrusil
article en
3 citations

Abstract

Despite the advances in automated medical image segmentation, AI models still underperform in various clinical settings, posing challenges for integration into real-world workflows. In this pre-registered prospective multicenter evaluation, we analyzed 20 state-of-the-art mandibular segmentation models across 19,218 segmentations of 1,000 clinically resampled CT/CBCT scans. Our results suggest that for a given model, segmentation accuracy can vary by up to 25% in Dice score as socio-technical factors such as voxel size, bone orientation, and patient conditions (e.g., osteosynthesis or pathology) shift from favorable to adverse. Higher sharpness, isotropic smaller voxels, and neutral orientation significantly improved results, while metallic osteosynthesis and anatomical complexity led to significant degradation. Our findings challenge the common view of AI models as “plug-and-play” tools and suggest evidence-based optimization recommendations for both clinicians and developers. This will in turn boost the integration of AI segmentation tools in routine healthcare.

Expert Systems with ApplicationsVol. 299
University Medical Center Groningen (NL), Radboud University Nijmegen (NL), Shanghai Jiao Tong University (CN), University of Basel (CH), Université Paris Cité (FR), University of Michigan (US), University Hospital of Basel (CH), University Medical Center (US), Radboud University Medical Center (NL), Humboldt-Universität zu Berlin (DE), Einstein Center Digital Future (DE), Sorbonne Paris Cité (FR), Communauté urbaine de Lyon (FR), UMIT - Private Universität für Gesundheitswissenschaften, Medizinische Informatik und Technik (AT), Abiomed (Germany) (DE), Essen University Hospital (DE), École d'Ingénieurs en Chimie et Sciences du Numérique (FR), University of Göttingen (DE), RWTH Aachen University (DE), Materialise (Belgium) (BE), KU Leuven (BE), University of Minho (PT)
Openalex Percentile: Top 5%
Radiomics and Machine Learning in Medical Imaging
4.08
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.