SegMambaDP: a long-range sequential modeling mamba network for 3D dose prediction in radiotherapy

Radiotherapy is a cornerstone of cancer treatment, and its therapeutic efficacy critically depends on the precision of the delivered dose distribution. The current clinical standard for treatment planning involves an inverse optimization process where physicists define clinical objectives and constraints, followed by computationally intensive dose calculations, often using methods like Monte Carlo simulation. While accurate, this iterative process is time-consuming, leading to delays and potential variability in plan quality. Deep learning methods have improved dose-prediction accuracy, but they often struggle to model fine-grained spatial relationships in complex anatomy. This study aims to develop a novel State Space Model (SSM)-based network for accurately predicting 3D radiotherapy dose distributions in head and neck (H&N) and cervical cancer. We introduce SegMambaDP, an SSM–based network for 3D dose prediction, and evaluate it on the OpenKBP dataset and an institutional cervical-cancer VMAT cohort. The network employs a cascading architecture that integrates a Tri-orientated Mamba (ToM) module to strengthen long-range dependency modeling of 3D features and a feature-level Uncertainty Estimation (FUE) module to enhance multi-scale feature fusion. Moreover, SegMambaDP incorporates both anatomical structure and beam angle information to boost clinical adaptability. Finally, model performance was assessed by comparison with existing methods using the Dose score, DVH score, and relevant dosimetric indicators using the 2020 AAPM OpenKBP Challenge dataset and our institution’s cervical cancer dataset. On the OpenKBP dataset, SegMambaDP achieved a Dose score of 2.398 ± 1.028 Gy and a DVH score of 1.254 ± 1.660 Gy. On the cervical cancer dataset, it achieved a Dose score of 2.428 ± 1.028 Gy and a DVH score of 1.274 ± 1.760 Gy. In both datasets, the predicted Conformity Index (CI) and Homogeneity Index (HI) for the Planning Target Volume (PTV) closely matched the true values, with an average prediction error ranging from 0.01 to 0.03(dimensionless). This study introduces a novel dose prediction model, SegMambaDP, designed to accurately forecast clinical dose distributions for head and neck and cervical cancer patients. SegMambaDP can offer a potential solution for dose distribution prediction in computer-assisted radiotherapy planning systems.