Dataset, Challenge, and Evaluation for Tumor Segmentation Variability

In numerous medical scenarios, segmenting clinical targets is highly subjective, influenced by the doctors' expertise and preferences, which results in significant multi-rater variability. This inherent annotation ambiguity poses a challenge for the practical deployment of data-driven techniques and raises concerns about the reliability of automatic predictions by medical artificial intelligence (AI) systems. To address this issue, we host a grand challenge (MMIS-2024) at ACM MM '24 to explore the problem of multi-rater medical image segmentation. First, we have released two datasets publicly, one on nasopharyngeal carcinoma (NPC) and the other on glioblastoma (GBM). For NPC, one challenge track encourages participants to develop models that utilize the four expert-provided labels per sample. The second GBM track explores the one-sample-one-label setting in the context of multi-rater segmentation. Here, different experts annotated different GBM samples for training. Finally, to assess the submissions, we employ two distinct sets of metrics, designed to evaluate prediction diversity and personalization, respectively. By exploring the two tasks with different metrics, the MMIS-2024 challenge aims to establish a global benchmark for multi-rater medical image segmentation, facilitating clinical AI deployments.