FmH2ST: foundation model-based spatial transcriptomics generation from histological images

Spatial transcriptomics (ST) reveals gene expression distributions within tissues. Yet, predicting spatial gene expression from histological images still faces the challenges of limited ST data that lack prior knowledge, and insufficient capturing of inter-slice heterogeneity and intra-slice complexity. To tackle these challenges, we introduce FmH2ST, a foundation model-based method for spatial gene expression prediction. Equipped with powerful foundation models pretrained on massive images, FmH2ST employs a dual-branch framework to integrate prior knowledge from foundation model and fine-grained details from spot images. The foundation model branch employs a multilevel feature extraction strategy to obtain enriched features with slice context for capturing inter-slice heterogeneity, and a dual-graph strategy to obtain spatial and semantic enriched features for modeling intra-slice complexity. The spot-specific learning branch leverages multiscale convolutions, Transformer, and graph neural network to extract fine-grained spot features. The outputs of two branches are adaptively fused for better prediction under a collaborative branch learning strategy. Experimental results show FmH2ST outperforms state-of-the-art methods on benchmark datasets. FmH2ST can denoise the raw gene expressions, reveal cancer spatial heterogeneity and gene co-expression patterns, and support the inference of gene regulatory networks. Overall, FmH2ST is effective for predicting ST, with potential applications in clinical diagnostics and personalized treatment.