Pathway-Aware Multimodal Transformer for Explainable Survival Prediction in Bladder Cancer
Clinical survival prognostication in Bladder Urothelial Carcinoma (TCGA-BLCA) is hindered by the translational disconnect between spatial histopathology and molecular genomics, where conventional models suffer from genomic overfitting or uninterpretable late-fusion concatenation. We present OncoBot, a Pathway-Aware Multimodal Transformer (PAMT) that unifies gigapixel Hematoxylin and Eosin (H&E) whole-slide images (WSIs) with bulk RNA-Seq transcriptomics. OncoBot regularizes 4,500 gene features into 320 canonical KEGG cancer pathways via a sparse bipartite binary mask, utilizing pathway vectors as Queries to guide asymmetric multi-head cross-attention over 384-dimensional morphological tokens extracted by a self-supervised DINO ViT-S/16 backbone across 1,024 tissue patches. An optimized preprocessing pipeline resolves OpenSlide C-level memory leaks through dynamic thumbnailing and controlled multiprocessing, slashing tiling latency from 120s to 15s while compressing 4GB WSIs into 1.5MB latent tensors. Optimized via a joint objective combining Cox Proportional Hazards and InfoNCE contrastive mutual information loss on 403 TCGA-BLCA patients, OncoBot achieves a state-of-the-art Harrell’s C-index of 0.784, outperforming unimodal transcriptomics (0.672), vision MIL (0.698), and late concatenation (0.718) by up to 16.7%. Finally, high-resolution spatial cross-attention heatmaps and an integrated 70B-parameter LLM clinical dossier generator translate multimodal risk predictions into interpretable, actionable insights for multidisciplinary tumor boards. Key Words: Multimodal deep learning, cross-attention transformer, gigapixel whole-slide imaging, computational pathology, KEGG biological pathways, cancer survival prediction, Cox proportional hazards, vision transformers, MLOps, explainable artificial intelligence, clinical decision support systems, TCGA-BLCA.
Authors
- Gowtham Raj M
- Bhuvan B
- Arun Chavan
- Charan Teja G S
- Charan Sai Tej K V
Institutions
- Visvesvaraya Technological University (IN)
Publication Details
- Journal
- International Scientific Journal of Engineering and Management
- Published
- 2026-11-21
- DOI
- https://doi.org/10.55041/isjem08764
- Primary Topic
- AI in cancer detection
- Type
- article
- Field-Weighted Citation Impact
- 0.00