DeCoST: unveiling cell type heterogeneity in spatial transcriptomics based on inter-domain alignment and Gaussian kernel conditional autoregressive

Spatial transcriptomics (STs) has emerged as a transformative approach to elucidate cellular heterogeneity and spatial organization within complex tissue microenvironments. However, the analysis of ST data is challenged by limited spatial resolution, resulting in mixed expression profiles at each spatial location. Moreover, the precious spatial information is rarely exploited, and noise issues in spatial transcriptomes (STs) are often overlooked by computational deconvolution methods. In this study, a novel computational framework for STs deconvolution (DeCoST), called DeCoST, is presented. DeCoST capitalizes on the valuable spatial context information by integrating a Gaussian kernel-based conditional autoregressive model. Additionally, the method employs domain adaptation techniques to address platform effects between single-cell and ST data, enabling robust cell type identification. Evaluations on simulated datasets under diverse spatial configurations, as well as real-world case studies on human pancreatic ductal adenocarcinoma, mouse olfactory bulb, and mouse brain samples, demonstrate the superior performance of DeCoST compared to existing deconvolution approaches. The method's ability to accurately map region-specific cell types and uncover spatial interactions advances our understanding of complex tissue organization and function, with broad applications in disease research and developmental biology.