STANCE: a unified statistical model to detect cell-type-specific spatially variable genes in spatial transcriptomics

One of the major challenges in spatial transcriptomics is to detect spatially variable genes (SVGs), whose expression patterns are non-random across tissue locations. Many SVGs correlate with cell type compositions, introducing the concept of cell type-specific SVGs (ctSVGs). Existing ctSVG detection methods treat cell type-specific spatial effects as fixed effects, leading to tissue spatial rotation-dependent results. Moreover, SVGs may exhibit random spatial patterns within cell types, meaning an SVG is not always a ctSVG, and vice versa, further complicating detection. We propose STANCE, a unified statistical model for both SVGs and ctSVGs detection under a linear mixed-effect model framework that integrates gene expression, spatial location, and cell type composition information. STANCE ensures tissue rotation-invariant results, with a two-stage approach: initial SVG/ctSVG detection followed by ctSVG-specific testing. We demonstrate its performance through extensive simulations and analyses of public datasets. Downstream analyses reveal STANCE's potential in spatial transcriptomics analysis.