A deep learning representation and spatial Bayesian cell-type deconvolution for spatial transcriptomics

Background Spatial transcriptomics provides unprecedented insights into gene expression in the spatial context of tissues; however, some mainstream techniques lack quantitative analysis of uncertainty. Methods In this study, we propose a computational framework—Spatial Deconvolution via Deep Gaussian Processes (SDDGP)—that integrates deep Gaussian processes with Bayesian inference for spatially aware deconvolution of spatial transcriptomics data. The framework consists of three tightly coupled components. A multi-layer deep Gaussian process (DGP) captures nonlinear spatial dependencies in cell-type composition by propagating spatial coordinates through successive Gaussian process (GP) layers, each equipped with sparse variational inducing points and a radial basis function (RBF) kernel that can be combined with an external Matern spatial kernel. The DGP output serves as an adaptive, spatially informed prior for per-spot cell-type proportions, which are then estimated via a Negative-Binomial Markov chain Monte Carlo (MCMC) sampler with automatic proposal tuning, thinning, and split-chain convergence diagnostics. Results We compared our method with existing approaches on two datasets: a pancreatic ductal adenocarcinoma (PDAC) dataset with matched single-cell RNA sequencing and spatial transcriptomics, and a human thymus dataset. Across both datasets, our method estimated cell type proportions more accurately than existing methods, particularly for rare cell types, and recovered biologically meaningful spatial patterns such as the distribution of T cells and plasma cells in PDAC. The Bayesian formulation also provides a quantification of uncertainty for each estimate, which aids the interpretation of spatial heterogeneity in complex tissues.