Performances of experts and automated methods on new multiple sclerosis lesions detection: insights from the MSSeg2 challenge

The assessment of new Multiple Sclerosis (MS) lesions is a time-consuming and difficult task that may lead to an underestimation of patient disease activity. Efficient methods to detect new T2/FLAIR lesions are therefore critical to assist clinicians in this process. In this context, we proposed and organized in 2021 the MSSeg2 challenge aiming at comparing methods segmenting new MS lesions. For this purpose, we built a high-quality dataset of 100 pairs of FLAIR MRI with precise delineation of new MS lesions with a size superior to 3 mm[Formula: see text]. From these 100 pairs of MS patient images from various scanners and French clinical centers, 40 were shared to the challengers before they submitted their methods. 30 methods from 24 international teams were submitted and evaluated on the FLI-IAM dedicated platform on the 60 remaining images. Overall, we observed that even at lesion scale, expert annotations were variable (40% of lesions were annotated by two or fewer experts out of the four). The best expert exhibited a mean F1 score of 0.679 (SD = 0.345) while the best method exhibited a mean F1 score of 0.698 (SD = 0.295) on the 35 patients with new lesions. Moreover, we did not observe evidence of differences between the top-ranked methods and the best expert performances as evaluated by the F1 score (9 methods exhibited no clear evidence against no difference of mean F1). Similarly, we did not observe evidence of difference in performances between the top-ranked methods and the best expert in classifying patients into a 3 categories clinically relevant scale (0 lesion, 1 or 2 lesions and >2 lesions; 21 methods exhibited no clear evidence against no difference of performances with a best classification accuracy of 85 % for both the best expert and the best method). While results from current automated methods still remain perfectible, our results highlight their potential usefulness in detecting new FLAIR MS lesions.