A novel multimodal framework integrating pathomics, deep learning, and machine learning for breast cancer histological grades classification

The present study is aimed at developing and testing a strong, interpretable, and multi-scale computational pathology framework for the automated histologic grade (grade-based) grading of invasive ductal carcinoma (IDC) from hematoxylin and eosin (H&E) whole-slide images (WSIs). A multi-center dataset consisting of 3,660 WSIs from 925 patients with IDC from five institutions was assembled and quality controls were very stringent. Images were captured at four magnifications (4X, 10X, 20X, 40X), in order to capture both a view of global tissue architecture and fine detail of the nuclei. Multi-modality features were extracted: low and high magnification deep features (DINOv2 ViT-S/16, ConvNeXt V2 Small), 71 handcrafted radiomics descriptors extracted from H&E channels and three pseudo-nuclei indices. Correlation filtering and four feature selection approaches (mutual information, recursive feature elimination, ANOVA, LASSO) were used on a modality per modality basis. An Attention MIL framework with a novel cross scale attention module and cross scale consistency loss was used for feature fusion and grading. Three strategies of fusion (early, late, hybrid) were compared. Model performance was evaluated using 5-folder cross-validation, internal, and external validation on an independent cohort (n = 922 WSIs). The hybrid fusion of LASSO selected features resulted in the best performance as the accuracy was found to be 92.5% (internal) and 90.9% (external) and the AUC were found to be 93.5% and 92.1% respectively. High magnitude deep features proved to be the most informative single modality and multi-modality fusion increased both the accuracy and robustness. The proposed framework was better than end-to-end baselines (p < 0.001), and the recall was balanced across all grades. Attention heatmaps identified regions of diagnostic interest which were consistent with pathologist annotations. The multi-scaled, multi modal Attention-MIL framework provides accurate, interpretable and generalizable grading of IDC, outperforming conventional and end-to-end methods. Its inter-institutional strength and biological interpretability make it an appealing candidate for a clinical decision-support tool.