Development and validation of a methodology to simulate interobserver variability

OBJECTIVE: Contouring for radiotherapy planning remains time intensive. As such, interobserver contour variation (IOV) studies are limited in patient and/or observer numbers (1). This study provides a methodology to simulate IOV on images containing a single contour. Approach. 14 gynaecological cancer radiotherapy target volumes delineated by five observers on T2W-MRI scans were used to develop an AI-based approach and three rule-based (simple, normal, adapted) approaches to generate estimated minimum and maximum (min/max) contours. The rule-based approaches used delineation uncertainty of the observers' contours. The AI-based approach combined seven nnU-Net models into an ensemble. The development of the AI-based and rule-based approaches utilised 7-fold and leave-one-out cross validation, respectively. Validation used a separate/second dataset with the same characteristics and six observers. A single contour was used as the basis and the remaining five contours were used to generate the min/max contours Performance of each approach was assessed using dice similarity coefficient (DSC), mean surface distance (MSD), surface DSC (sDSC) and Hausdorff distance (HD). The best approach was utilised in a pipeline to generate a min/max contour. Multiple contours were generated between these by applying non-uniform expansions and contractions to a single contour. The comparison metrics of the observers' and simulated contours were computed in pairwise, with a mean and standard deviation calculated. Main Results. The adapted rule-based approach performed best across all metrics and was used in the subsequent pipeline. The AI-based approach had comparative median values to the rule-based approaches, but a much wider range of values. The observers' and simulated contours had a DSC of 0.77±0.13 and 0.78±0.20, respectively and an MSD of 2.4±1.7 mm and 2.2±1.5 mm, respectively. Significance. This study provides a methodology to generate additional observer contours from a single observer contour based on IOV data.

Authors

Institutions

Publication Details

Journal
Biomedical Physics & Engineering Express
Published
2026-11-17
DOI
https://doi.org/10.1088/2057-1976/aec2f2
Primary Topic
Advanced Radiotherapy Techniques
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Development and validation of a methodology to simulate interobserver variability

Rhianna Brown, Amy Walker, Karen Lim, Lois Charlotte Holloway et al.
4 links
Biomedical Physics & Engineering Express
Advanced Radiotherapy Techniques
article

Development and validation of a methodology to simulate interobserver variability

Rhianna Brown, Amy Walker, Karen Lim, Lois Charlotte Holloway, Peter E. Metcalfe, Dean L. Cutajar, Phillip Chlap
article en

Abstract

OBJECTIVE: Contouring for radiotherapy planning remains time intensive. As such, interobserver contour variation (IOV) studies are limited in patient and/or observer numbers (1). This study provides a methodology to simulate IOV on images containing a single contour. Approach. 14 gynaecological cancer radiotherapy target volumes delineated by five observers on T2W-MRI scans were used to develop an AI-based approach and three rule-based (simple, normal, adapted) approaches to generate estimated minimum and maximum (min/max) contours. The rule-based approaches used delineation uncertainty of the observers' contours. The AI-based approach combined seven nnU-Net models into an ensemble. The development of the AI-based and rule-based approaches utilised 7-fold and leave-one-out cross validation, respectively. Validation used a separate/second dataset with the same characteristics and six observers. A single contour was used as the basis and the remaining five contours were used to generate the min/max contours Performance of each approach was assessed using dice similarity coefficient (DSC), mean surface distance (MSD), surface DSC (sDSC) and Hausdorff distance (HD). The best approach was utilised in a pipeline to generate a min/max contour. Multiple contours were generated between these by applying non-uniform expansions and contractions to a single contour. The comparison metrics of the observers' and simulated contours were computed in pairwise, with a mean and standard deviation calculated. Main Results. The adapted rule-based approach performed best across all metrics and was used in the subsequent pipeline. The AI-based approach had comparative median values to the rule-based approaches, but a much wider range of values. The observers' and simulated contours had a DSC of 0.77±0.13 and 0.78±0.20, respectively and an MSD of 2.4±1.7 mm and 2.2±1.5 mm, respectively. Significance. This study provides a methodology to generate additional observer contours from a single observer contour based on IOV data.

Biomedical Physics & Engineering Express
University of Wollongong (AU), Ingham Institute (AU)
Good health and well-being
Advanced Radiotherapy Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon