A unified framework for identification of cell-type-specific spatially variable genes in spatial transcriptomic studies

Characterizing cell-type-specific spatially variable genes (SVGs) within tissue context is essential for exploring the landscape of complex biological systems in spatial transcriptomic (ST) studies. In this paper, we present a unified framework, the Mixture of Mixed Models (MMM), designed to directly model RNA count data and identify cell-type-specific SVGs while accounting for cell type composition and correcting for platform effects. Through a comprehensive simulation study and the analyses of eight publicly available ST datasets from various tissues and technologies with different resolutions, we demonstrate the effectiveness and robustness of MMM in identifying cell-type-specific SVGs. Notably, our integrative analysis with genome-wide association studies reveals that the cell-type-specific SVGs identified by MMM in a mouse brain study exhibit significant heritability enrichment in brain-related phenotypes. This finding suggests that cell-type-specific SVGs play a vital role in elucidating the mechanisms underlying complex traits and diseases. When applying MMM to analyze a high-resolution Xenium human breast cancer dataset by accounting for the uncertainties in cell segmentation, we find that certain cell-type-specific SVGs may contribute to cell-cell communications, thereby regulating the tissue microenvironment. Furthermore, we show the versatility of MMM by applying it to the 3D tissue models constructed from multiple ST slices, highlighting its utility in analyzing the 3D ST data.