An explainable longitudinal multi-modal fusion model for predicting neoadjuvant therapy response in women with breast cancer

Multi-modal image analysis using deep learning (DL) lays the foundation for neoadjuvant treatment (NAT) response monitoring. However, existing methods prioritize extracting multi-modal features to enhance predictive performance, with limited consideration on real-world clinical applicability, particularly in longitudinal NAT scenarios with multi-modal data. Here, we propose the Multi-modal Response Prediction (MRP) system, designed to mimic real-world physician assessments of NAT responses in breast cancer. To enhance feasibility, MRP integrates cross-modal knowledge mining and temporal information embedding strategy to handle missing modalities and remain less affected by different NAT settings. We validated MRP through multi-center studies and multinational reader studies. MRP exhibited comparable robustness to breast radiologists, outperforming humans in predicting pathological complete response in the Pre-NAT phase (ΔAUROC 14% and 10% on in-house and external datasets, respectively). Furthermore, we assessed MRP's clinical utility impact on treatment decision-making. MRP may have profound implications for enrolment into NAT trials and determining surgery extensiveness.

Authors

Institutions

Publication Details

Journal
Nature Communications
Published
2024-11-07
DOI
https://doi.org/10.1038/s41467-024-53450-8
Citations
53
Primary Topic
Breast Cancer Treatment Studies
Type
article
Field-Weighted Citation Impact
15.11

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

An explainable longitudinal multi-modal fusion model for predicting neoadjuvant therapy response in women with breast cancer

Luyi Han, Muzhen He, Yuan Gao, Hugo M. Horlings et al.
53 citations
Nature Communications
Breast Cancer Treatment Studies
15.11
article

An explainable longitudinal multi-modal fusion model for predicting neoadjuvant therapy response in women with breast cancer

Luyi Han, Muzhen He, Yuan Gao, Hugo M. Horlings, Valentina Laurita Longo, Anna D’Angelo, Xin Wang, Sofía Ventura‐Díaz, Marleen Kok, Ritse M. Mann, Frederieke van Duijnhoven, Xiao‐Mei Li, Tianyu Zhang, Hong-Yu Zhou, Tao Tan, Zeyan Xu, Regina G. H. Beets‐Tan, Jonas Teuwen, Zaiyi Liu, Alva B. Weir
article en
53 citations

Abstract

Multi-modal image analysis using deep learning (DL) lays the foundation for neoadjuvant treatment (NAT) response monitoring. However, existing methods prioritize extracting multi-modal features to enhance predictive performance, with limited consideration on real-world clinical applicability, particularly in longitudinal NAT scenarios with multi-modal data. Here, we propose the Multi-modal Response Prediction (MRP) system, designed to mimic real-world physician assessments of NAT responses in breast cancer. To enhance feasibility, MRP integrates cross-modal knowledge mining and temporal information embedding strategy to handle missing modalities and remain less affected by different NAT settings. We validated MRP through multi-center studies and multinational reader studies. MRP exhibited comparable robustness to breast radiologists, outperforming humans in predicting pathological complete response in the Pre-NAT phase (ΔAUROC 14% and 10% on in-house and external datasets, respectively). Furthermore, we assessed MRP's clinical utility impact on treatment decision-making. MRP may have profound implications for enrolment into NAT trials and determining surgery extensiveness.

Nature CommunicationsVol. 15(1)
Fujian Medical University (CN), Harvard University (US), Radboud University Nijmegen (NL), Jinan University (CN), Kunming Medical University (CN), Maastricht University Medical Centre (NL), Cork University Hospital (IE), Agostino Gemelli University Polyclinic (IT), Radboud University Medical Center (NL), The Netherlands Cancer Institute (NL), Key Laboratory of Guangdong Province (CN), Fujian Provincial Hospital (CN), Guangdong Academy of Medical Sciences (CN), Istituti di Ricovero e Cura a Carattere Scientifico (IT), Guangdong Provincial People's Hospital (CN), Macao Polytechnic University (MO), Southern Medical University (CN), St. Joseph’s Healthcare Hamilton (CA), Fuzhou University (CN)
China Scholarship Council
Openalex Percentile: Top 1%
Breast Cancer Treatment Studies
15.11
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.