Modality-Aware Distillation Network for Microvascular Invasion Prediction of Hepatocellar Carcinoma From MRI Images

Microvascular invasion (MVI) of hepatocellular carcinoma (HCC) is a crucial histopathologic prognostic factor associated with cancer recurrence after liver transplantation or hepatectomy. Recently, clinicoradiologic characteristics are combined with medical images to enhance the HCC prediction. However, compared to medical imaging data, the clinicoradiologic characteristics (e.g., APOe4 genotyping) is not easy to collect or even unavailable, as it requires more efforts of clinicians and more medical instruments for collecting diverse measurements. This work explores how to transfer the knowledge of a teacher network learned from non-image clinical data and image data to a student network with only image data such that the student network can leverage the transferred clinical information to boost HCC classification with only imaging data as input. Specifically, we present a modality-aware distillation network (MD-Net) to transform non-image clinicoradiologic from the teacher network to the student network. The teacher network integrates non-image clinicoradiologic characteristics with two 3D MRI modality images via two MRI-clinical-fusion modules and a symmetric attention (SA) module, while the student network extracts features from two modality MRI data via two MRI-only modules and then refine these two MRI features via a SA module. A classification-level distillation and a feature-level distillation are jointly utilized to transfer the clinical information between teacher and student networks. Furthermore, we design a novel self-supervised task to predict clinicoradiologic characteristics from the imaging data to further enhance the downstream HCC classification. The experimental results from our collected dataset and a multi-modal sarcasm detection dataset have demonstrated the effectiveness of our approach. Specifically, we achieved an AUC score of 71.86% and 75.51% respectively, surpassing the performance of the state-of-the-art classification methods.