Pathway-Aware Multimodal Transformer for Explainable Survival Prediction in Bladder Cancer

Clinical survival prognostication in Bladder Urothelial Carcinoma (TCGA-BLCA) is hindered by the translational disconnect between spatial histopathology and molecular genomics, where conventional models suffer from genomic overfitting or uninterpretable late-fusion concatenation. We present OncoBot, a Pathway-Aware Multimodal Transformer (PAMT) that unifies gigapixel Hematoxylin and Eosin (H&E) whole-slide images (WSIs) with bulk RNA-Seq transcriptomics. OncoBot regularizes 4,500 gene features into 320 canonical KEGG cancer pathways via a sparse bipartite binary mask, utilizing pathway vectors as Queries to guide asymmetric multi-head cross-attention over 384-dimensional morphological tokens extracted by a self-supervised DINO ViT-S/16 backbone across 1,024 tissue patches. An optimized preprocessing pipeline resolves OpenSlide C-level memory leaks through dynamic thumbnailing and controlled multiprocessing, slashing tiling latency from 120s to 15s while compressing 4GB WSIs into 1.5MB latent tensors. Optimized via a joint objective combining Cox Proportional Hazards and InfoNCE contrastive mutual information loss on 403 TCGA-BLCA patients, OncoBot achieves a state-of-the-art Harrell’s C-index of 0.784, outperforming unimodal transcriptomics (0.672), vision MIL (0.698), and late concatenation (0.718) by up to 16.7%. Finally, high-resolution spatial cross-attention heatmaps and an integrated 70B-parameter LLM clinical dossier generator translate multimodal risk predictions into interpretable, actionable insights for multidisciplinary tumor boards. Key Words: Multimodal deep learning, cross-attention transformer, gigapixel whole-slide imaging, computational pathology, KEGG biological pathways, cancer survival prediction, Cox proportional hazards, vision transformers, MLOps, explainable artificial intelligence, clinical decision support systems, TCGA-BLCA.

Authors

Institutions

Publication Details

Journal
International Scientific Journal of Engineering and Management
Published
2026-11-21
DOI
https://doi.org/10.55041/isjem08764
Primary Topic
AI in cancer detection
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Pathway-Aware Multimodal Transformer for Explainable Survival Prediction in Bladder Cancer

Gowtham Raj M, Bhuvan B, Arun Chavan, Charan Teja G S et al.
International Scientific Journal of Engineering and Management
AI in cancer detection
article

Pathway-Aware Multimodal Transformer for Explainable Survival Prediction in Bladder Cancer

Gowtham Raj M, Bhuvan B, Arun Chavan, Charan Teja G S, Charan Sai Tej K V
article en

Abstract

Clinical survival prognostication in Bladder Urothelial Carcinoma (TCGA-BLCA) is hindered by the translational disconnect between spatial histopathology and molecular genomics, where conventional models suffer from genomic overfitting or uninterpretable late-fusion concatenation. We present OncoBot, a Pathway-Aware Multimodal Transformer (PAMT) that unifies gigapixel Hematoxylin and Eosin (H&E) whole-slide images (WSIs) with bulk RNA-Seq transcriptomics. OncoBot regularizes 4,500 gene features into 320 canonical KEGG cancer pathways via a sparse bipartite binary mask, utilizing pathway vectors as Queries to guide asymmetric multi-head cross-attention over 384-dimensional morphological tokens extracted by a self-supervised DINO ViT-S/16 backbone across 1,024 tissue patches. An optimized preprocessing pipeline resolves OpenSlide C-level memory leaks through dynamic thumbnailing and controlled multiprocessing, slashing tiling latency from 120s to 15s while compressing 4GB WSIs into 1.5MB latent tensors. Optimized via a joint objective combining Cox Proportional Hazards and InfoNCE contrastive mutual information loss on 403 TCGA-BLCA patients, OncoBot achieves a state-of-the-art Harrell’s C-index of 0.784, outperforming unimodal transcriptomics (0.672), vision MIL (0.698), and late concatenation (0.718) by up to 16.7%. Finally, high-resolution spatial cross-attention heatmaps and an integrated 70B-parameter LLM clinical dossier generator translate multimodal risk predictions into interpretable, actionable insights for multidisciplinary tumor boards. Key Words: Multimodal deep learning, cross-attention transformer, gigapixel whole-slide imaging, computational pathology, KEGG biological pathways, cancer survival prediction, Cox proportional hazards, vision transformers, MLOps, explainable artificial intelligence, clinical decision support systems, TCGA-BLCA.

International Scientific Journal of Engineering and ManagementVol. 05(11)
Visvesvaraya Technological University (IN)
AI in cancer detection
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon