HMT: A Hybrid Multimodal Transformer With Multitask Learning for Survival Prediction in Head and Neck Cancer

Survival prediction is crucial for cancer patients as it offers prognostic information for treatment planning. Recently, deep learning-based multimodal survival prediction models have demonstrated promising performance. However, current models face challenges in effectively utilizing heterogeneous multimodal data (e.g., positron emission tomography (PET)/computed tomography (CT) images and clinical tabular) and extracting essential information from tumor regions, resulting in suboptimal survival prediction accuracy. To tackle these limitations, in this article, we propose a novel hybrid multimodal transformer model (HMT), namely HMT, for survival prediction from PET/CT images and clinical tabular in Head and Neck (H&N) cancer. Specifically, we develop hybrid attention modules to capture intramodal information and intermodal correlations from multimodal PET/CT images. Moreover, we design hierarchical Tabular Affine transformation modules (TATMs) to integrate supplementary insights from the heterogenous tabular with images via affine transformations. The TATM dynamically emphasizes features contributing to the survival prediction while suppressing irrelevant ones during integration. To achieve finer feature fusion, TATMs are hierarchically embedded into the network, allowing for consistent interaction between tabular and multimodal image features across multiple scales. To mitigate interferences caused by irrelevant information, we introduce tumor segmentation as an auxiliary task to capture features related to tumor regions, thus enhancing prediction accuracy. Experiments demonstrate our superior performance. The code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/gluucose/HMT</uri>.